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ABSTRACT 


This  report  describes  the  current  practices  and  evaluative  aspects  of 
human  performance  assessment  in  Air  Force  Systems,  The  human  performance 
test  programs  for  thirty-four  systems  and  subsystems  representing  the  major 
types  of  systems  (aeronautical,  electronic,  missile,  and  space)  used  by  the 
Air  Force  are  reviewed.  For  these  systems,  the  major  functional  areas  covered 
include;  (1)  Air  Force  policies,  directives,  requirements,  and  constraints 
concerning  the  development  and  assessment  of  system  tests  and  human  per¬ 
formance;  (2)  the  behavioral  sciences  approach  to.  and  technology  for, 
assessing  human  performance;  and  (3)  Air  Force  practices  in  assessment  of 
human  performance.  Throughout,  the  systems  context,  within  which  human 
performance  is  conceived  and  evaluated,  is  emphasized.  Consequently,  the 
techniques  within  the  behavioral  sciences  for  examining  human  performance 
conceptually  and  empirically  in  the  system  test  environment  is  a  particularly 
practicable  part  of  the  report.  The  report  is  supported  by  many  useful  tables 
and  charts ,  excerpts  from  teat  directives  pertinent  to  human  performance 
assessment.,  and  approximately  600  categorized  references. 
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PRACTICES  IN  THE  ASSESSMENT  OF  HUMAN 
PERFORMANCE  IN  AIR  FORCE  SYSTEMS 

I 

OVERVIEW,  PROCEDURES  AND  FRAMEWORK  OF  THE  REVIEW 


A.  Overview  of  the  Report 


1.  Objectives 

This  report  was  written  as  an  interim  review,  a  progress  report,  of 
findings  concerning  methods  techniques,  and  practices  for  assessing  human 
performance.  It  is  part  of  a  two -phase  l'esearch  program  for  the  development 
of  a  technical  handbook  for  measuring  and  evaluating  human  performance  iu 
Air  Force  systems. 

2.  Perspective 

Human  performance  is  regarded  as  an  interaction  unit,  the  P.  E,  A. , 
that  is,  personnel  (P)  in  the  system  environment  (E)  (which  includes  other 
personnel,  equipment  and  ambient  conditions)  carrying  out  so:  mission- 
oriented  activity  (A).  Systems  are  defined  as  organizations  of  such  interactions. 
These  nP.  E.  A.  !l  interactions  are  postulated  as  the  fecal  units  of  system  and 
human  performance  assessment. 

The  assessment  process  is  visualized  in  two  or  three  general  steps: 
analysis  of  needs  for  information  concerning  human  performance,  acquisition 
of  additional,  empirical  information  concerning  performance,  and  evaluation 
of  the  data  in  terms  of  the  need/use  for  it.  There  are  needs  and  uses  for 
evaluative  information  continuously  during  the  four  life  stages  of  Air  Fo^ce 
systems.  A  large  body  of  requirements -documents  is  available  to  structure 
the  development  and  systematic  application  of  this  information  to  system  de¬ 
sign,  test,  and  evaluation.  A  variety  of  practices  are  currently  used  in  the 
description  and  evaluation  of  human  performance  in  the  systems  of  the  l,  S, 

Air  Force. 

3.  Content 


This  report  discusses  three  major  topics:  (1)  Air  Force  requirements 
concerning  the  development  and  assessment  of  system  and  human  performance, 

(2)  the  methodology  and  technology  of  the  behavioral  sciences  for  assessing 
human  performance,  and  (3)  current  uses  of  this  technology  in  Air  Force  systems 
as  illustrated  by  a  review  of  available  information  concerning  test  programs  in 
selected  systems.  Reference  and  source  reports  used  in  this  review  are 
appended. 


4*  Limitations 


The  discussions  herein  are  based  upon  (1)  the  perspectives  (as 
summarised  briefly  in  this  section)  of  the  technical  handbook  for  the  assess¬ 
ment  of  human  performance,  and  (?*)  reference  material  made  availaole  to 
us  during  the  study  for  selected  Air  Force  test  activities.  This  effort, 
strictly  speaking,  neither  evaluates  what  information  is  available  nor  pur¬ 
ports  to  cover  all  activities  and  techniques  (used  or  available  for  use)  in 
the  assessment  of  human  performance. 

The  opinions  concerning  the  focus  and  conceptual  aspects  of  assess¬ 
ing  performance  are  those  of  the  authors,  and  do  not  necessarily  report 
the  thinking  of  the  scientists  consulted  or  the  studies  reviewed. 

B«  Procedures  of  the  Review 

1 .  Sources  of  Information 

Several  sources  of  information  were  used  in  this  study.  These 

were: 

a.  U.S.  Air  Force  requirements- documents  concerning  the 
development  and  management  of  man- machine  systems. 

b.  Behavioral  science  and  systems  engineering  studies  method¬ 
ologically  related  to  man-machine  performance  assessment. 

c.  Tesf*  and  evaluation  planning  and  test  report  documents  for 
selected  Air  Force  systems. 

d.  Cognizant  personnel,  military  and  civilian,  active  in  the 
management,  development,  and  testing  of  man- machine 
systems  and  subsystems. 

Approximately  six  hundred  documents,  studies,  and  methodological 
reports  were  reviewed  during  this  study.  These  reports  varied  with 
respect  to  the  human  performance  being  investigated,  the  type  of  military 
system  and  its  stage  of  development,  the  approach  as  well  as  the  disciplines 
represented  in  the  investigating  team,  and  in  the  goals  of  the  specific 
assessment  project. 

Conferences  and  discussions  were  held  with  approximately  sixty  mili¬ 
tary  and  civilian  scientists  at  various  Air  Force  and  Contractor  locations 
engaged  in  man -machine  system  development  and  evaluation.  These 
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diseufcsiors  generally  were  structured  toward  the  information  and  clari¬ 
fications  deemed  necessary  to  reach  a  refined,  sophisticated,  and  yet 
practical  assessment  methodology  as  well  as  to  survey  current  practices 
in  measuring  human  performance.  Our  own  experiences  with  system  and 
human  performance  evaluations  influenced  the  discussion  of  other  scien¬ 
tist’s  experiences.  During  this  phase,  the  conceptual  approach  of  the 
technical  handbook  was  examined  and  iterated,  and  progressive  changes  were 
made  to  reflect  the  state  of  the  art  in  the  assessment  of  human  performance. 
The  handbook  will  profit  from  the  information  derived  during  this  study: 
this  report  anticipates  the  perspectives  and  emphasis  of  the  technical  hand¬ 
book, 

2,  Review  of  Requirements  for  Assessing  Human 
Performance  in  Air  Force  Sys terns 


The  policies  and  directives  of  the  Air  Force  concerning  the  man¬ 
agement,  development,  and  testing  of  military  systems  were  reviewed  sys¬ 
tematically  during  this  study.  The  major  sources  and  types  of  such  infor¬ 
mation  a  ^e  listed  below. 

,  Air  Force  Regulations 

.  Air  Force  Manuals 

,  Air  Force  Specification  Bulletins 

.  Air  Force  Systems  Command  Regulations 

.  Air  Force  Systems  Command  Manuals 

,  Air  Force  Systems  Command  Program  Management  Instructions 
,  Air  Force  Systems  Command  Division  Exhibits 
,  Military  and  Federal  Standards 

.  Military  Specifications 

In  general,  documents  were  reviewed  which  represent  current  re¬ 
quirements  for  developing  Air  Force  systems,  for  developing  the  personnel 
subsystems  of  those  systems,  for  testing  Air  Force  systems,  subsystems, 
an!  equipment,  and  for  testing  the  per  some]  subsystem  in  particular. 

3 .  Review  of  Illustrative  Methodologies  for  assessing  Human 
Performance  in  Air  Force  Systems 


Having  established,  by  way  of  the  revi<nv  of  requirements  doruments, 
the  context  for  assessing  human  performance,  a  search  was  then  made  of 
techniques  illustrative  of  current  t&chniqut  u  and  practices  in  the  behavioral 
sciences  and  in  the  development  of  Air  Force  systems,  relating  to  the  assess¬ 
ment  of  human  performance. 
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a.  Review  ol  Bohavloyal  Sciences  Technology 


From  the  methodological  viewpoint,  the  behavioral  sciences 
offer  useful  ways  of  conceiving  about  man- machine  systems,  and  of  col¬ 
lecting  empirical  information  and  drawing  conclusions  about  human  per¬ 
formance,  Much  of  this  technology  has,  of  course,  been  brought  to  bear 
on  military  and  other  governmental  projects.  It  is  likely,  however,  that 
there  are  new  techniques  under  development  and/or  old  techniques  thus 
far  relatively  untried  which  are  of  potential  use  in  the  assessment  of  human 
performance.  And  there  may  be  fresh  perspectives  on  the  application  of  the 
behavioral  sciences  to  the  task  of  assessing  human  performance,  particularly 
in  the  early  stages  of  system  life,  that  require  statement.  Accordingly, 
a  systematic  review  of  the  behavioral  science  technology  was  onducted, 
following  a  mapping  adapted  from  Barmack  (  3  ).  Boohs  and  periodical 
literature,  as  well  as  Government- sponsored  research  studies,  were  sought 
within  this  schema  of  the  fields  and  subfields  of  the  behavioral  sciences. 

The  schema  itself  is  presented  xn  Appendix  I.  Four  major  fields,  and  the 
basic  and  applied  research  subfields  within  each,  were  used  to  VUJL  V* 

the  survey  of  the  behavioral  science  literature. 

The  first  major  field  was  named  human  performance  studies 
and  was  broken  down  into  two  basic  research  subfields,  namely,  individual 
performance,  and  group  performance.  Some  twenty- six  topics  were  allo¬ 
cated  between  these  basic  research  subfields.  Sim  larly,  applied  research 
concerning  human  performance  was  divided  into  two  subfields,  human  engi¬ 
neering,  and  team- system  performance.  Eighteen  study  areas  were  covered 
in  the  latter  two  subfiexds. 

Three  other  major  research  fields  were  identified  as  related 
to  the  study  objectives.  These  were  personnel  research  studies,  human 
support  and  maintenance  studies,  and  economic  analysis  and  management 
activities.  Personnel  research  studies  were  expected  to  illustrate  methods 
for  relating  system  and  human  performance  to  individual  and  group  skills, 
training  and  to  training  curricula,  etc.  Studies  in  the  field  of  human  support 
were  expected  to  illustrate  techniques  for  studying  performance  as  a  function 
of  life  support,  equipment,  shelter  and  environmental  variables.  Finally, 
economics,  considered  as  a  behavioral  science,  waj  expected  to  yield  ways 
of  describing  and  diagramming  human  performance  and  of  renting  it  to  the 
cost-effectiveness  and  value  of  the  system. 
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Review  of  Air  Force  Systems 


Thirty-four  Air  Force  systems  and  related  systems  and  sub¬ 
systems*  were  reviewed  during  this  study.  These  man-machine  systems 
were  selected  to  provide: 

1)  Representation  of  the  major  types  of  systems  under 
development  and  test  by  Air  Force  Systems  Command, 

£}  Repxesentation  of  current  and  relatively  recent  systems 
within  each  type  of  system, 

3)  Overview  of  plans,  tests,  analyses,  and  reports  illus¬ 
trative  of  techniques  for  investigating  human  perform¬ 
ance  as  a  function  of  interfaces  w'th  system  equipment, 
personnel,  procedures,  and/or  environment, 

4)  Illustration,  as  far  as  possible,  of  test  and  evaluation 
activities  during  the  conceptual,  definition,  acquisition, 
and  operational  stages  of  system  life. 

Table  I  summarises  ♦'he  types  of  systems  and  specific  systems 
within  each  type  on  which  informat  on  was  collected  during  this  study.  For 
the  most  part,  the  designation  used  for  a  system  depended  not  so  much  on 
the  equipment  characteristics  of  the  system  (since,  from  this  point  of  view, 
some  systems  would  not  be  different  from  others)  as  upon  the  name  of  the 
division  of  Air  Force  Systems  Command  which  has  the  responsibility  for 
acquiring  the  system. 

C.  Conceptual  Framework  for  tnis  Review 

The  information  that  was  collected  in  this  study  and  the  way  in  which  it 
was  organized  was  affected  by  a  priori  perspectives  on  what  human  perform¬ 
ance  means  and  the  development  and  evaluation  of  human  performance  in 
Air  Force  systems.  These  perspectives  are  discussed  briefly  here,  together 
with  the  resulting  expectations  for  the  methodology  of  human  performance 
assessment, 

1 .  Considerations  Concerning  Human  Performance 

Operationally  considered,  human  performance  is  the  interaction  of 
system  personnel,  equipment,  environmental  conditions,  and  procedures. 


*N.A,  S.A.  projects  and  some  subsystems  within  Air  Force  system**  were 
also  reviewed  during  this  study. 


Table  £,  Systems,  Subsystems  and  Programs 
Reviewed  During  This  Study® 


Subsystem^  or  Program*5 

Aeronautical 

Mi*  Tile 

Electronic 

Gemini  B 

B-47  (1952) 

WS-315A  -  Thor 

425L  (1958  59) 

(1962-64) 

B-25  (1953-  54) 

1RBM  (1958-59) 

416L  (1959-61) 

Satellite 

SM-  6 d  -  Titan  I 

Control 

AN/A.PQ-24 

ICBM  (1958-63) 

465 L (1960-64) 

Facility 

(1964) 

(1954) 

WS-107A-1  -  Atlas 

480L  (1960) 

B-52  (1960-61) 

ICBM  (1959-61) 

Apollo 

473 L  (19«'.l) 

(1964-65) 

B-58  (1960-61) 

TM-  76 B  -  Mace 
(I960) 

412L  (1961-63) 

Manned 

Helicopter 

Orbiting 

Flight  Simula- 

WS-107C  -  Titan  II 

433L  (1962-63) 

Laboratory 

(1964-65) 

tor  (1962) 

ICBM  (1963-64) 

431L/482L 

CH-3C  Heli¬ 

Program  279 

(1962-64) 

copter  (1963) 

(1963) 

416M  (1963-64) 

C- 141 (1963-64) 

WS-  133  A,  B 

Minuteman  ICBM 

4831.  (1963) 

F-4C  (1964) 

(1962-65) 

466L  (1963) 

F-5A/B  (1964) 

496L  (1963) 

D.  O.  R.  A. 

F-’UA/B 

(1965) 

including  time  period  of  reported  human  performance  studies. 

°Names  ani  descriptions  of  these  system*  are  presented  in  Section  IV, 
pages  62  through  88. 
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Human  performance  is  circumscribed  and  directed  by  the  mission  of  the 
system*  A  system  is  an  organisation  of  interactions  oriented  toward  the 
accomplishment  ox  a  specified  goal  or  set  ox  goals. 

Studying  human  performance  in  systems  requires  detailed  infor¬ 
mation  concerning  the  interactions  among  human  components  (system 
personnel),  system  equipment  (e,  g  ,  displays,  controls,  test  points), 
environmental  conditions  (e.g*,  ambient  lighting,  noise,  temperature), 
and  activities  (system  procedures),  *  The  assessment  of  human  perform¬ 
ance  involves  the  comparison  of  what  is  known  about  these  various  inter¬ 
actions  with  what  is  expected  or  required  of  the  same  interactions  by  the 
mission  of  the  system. 

Figure  1  presents  a  useful  nummary  or  model  of  the  functioning  of 
the  individual  humrn  component  in  relating  to  the  system  environment* 

It  also  summarises  the  aspects  of  human  performance,  some,  or  many  of 
which,  were  expected  to  be  under  investigation  in  the  systems  surveyed 
and/or  the  literature  reviewed, 

2„  Considerations  Concerning  an  Overall  Assessment  Methodology 

A  generalized  assessment  process,  schematized  in  Figure  2,  was 
used  as  a  frame  of  reference  in  surveying  and  organizing  the  various  activi¬ 
ties  involved  in  the  evaluation  of  human  performance.  Accordingly,  it  was 
expected  that  the  assessment  of  human  performance  would  proceed  in  th  *ee 
phases*  The  first  of  these  phases  generally  includes  the  identification  of  a 
specific  need  for  information  concerning  some  aspect  of  human  perform ince 
and  the  setting  about  to  acquire  the  information.  The  need  for  informat, on 
varies  from  system  to  system  and  from  one  development  stage  the  next, 
but  typically  it  is  related  to  decisions  regarding  automation,  equipment, 
design,  etc.  What  is  usually  required  is  an  estimate  of  some  aspect  ox  per 
formance,  that  is,  information  predicting  th^  characteristic  of  one  inter¬ 
action  or  another  .nvolving  system  personnel.  Typically,  this  results  in 
setting  up  a  formal,  empirical  situation  in  which  the  predictions  and  expec¬ 
tations  concerning  performance  can  be  examined  under  specified  conditions. 
This  phase  of  the  assessment  process  can  be  expected  to  include  a  variety  of 
techniques  and  tools  for  describing  and  predicting  performance,  and  for 
planning  or  selecting  test  situations  designed  tu  provide  the  information. 


^Human  performance  maybe  described  functionally  \s  a  "PEA11  'nter  actio**, 
that  is,  personnel  (P),  in  an  environment  (E),  which  inciades  other  person¬ 
nel*  equipment  and  ambient  conditions,  carrying  out  some  activity  (A). 
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The  second  phase  of  the  hypothesized  assessment  process  includes 
the  conduct  of  the  tests  planned  in  phase  one  and  the  use  of  assorted  tech¬ 
niques  for  collecting  data.  These  techniques  are  variations  of  indirect  or 
direct  human  observation.  A  variety  of  test  situations  can  be  set  up  to  pro¬ 
vide  for  the  observation  of  human  performance,  e,  g.  ,  simulations,  field 
trials.  Human  observation  of  human  performance  is  usually  structured  and 
programmed,  instrumented  by  the  use  of  checklists,  rating  scales,  photog¬ 
raphy,  recorders,  etc.  This  phase  also  includes  the  reduction  and  analysis 
of  information  and  the  use  of  descriptive  arid  inferential  statistics,  expert 
judgment,  etc. 

The  preparation  of  test  reports  in  diverse  formats  and  the  use  of 
the  test  information  occur  in  the  third  phase  of  the  hypothesized  assessment 
pi^cess. 

3 ,  Considerations  Concerning  Need  for  Assessment  During 
System  Development 

This  study  was  principally  concerned  with  the  developing  Air  Force 
system  and  the  needs  for  and  accomplishments  in  assessing  human  perform¬ 
ance  as  the  system  grows.  The  generalized  system  growth  process  may  be 
thought  to  consist  of  four  stages: 

.  ittage  One- -requirements  determination 

•  Stage  Two- -determination  of  design  consequences 

.  Stage  Three- -design  and  integration  of  the  system 

.  Stage  Four- -formal  test  and  evaluation  prior  to 

operational  use 

The  process  is  iterative,  a  series  of  approximations  in  which  pre¬ 
dictions  concerning  human  performance  are  intimately  related  to  various 
systems  engineering  activities  and  decisions.  This  perspective,  detailed  in 
Table  II,  was  the  context  of  our  inquiry  into  techniques  used  by  systems  and 
human  factors  engineering  personnel  in  co t'cei\ing  about  and  dealing  with 
human  performance  during  the  stages,  particularly  the  early  stages,  of 
system  development. 

D.  Organization  of  this  Report 

This  report  is  divided  into  five  sections  supported  by  appendix  and 
reference  materials. 

Section  I  introduces  the  report  describes  the  study  per¬ 
spectives,  procedures,  framework  and  limitations. 
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Section  II  contains  a  summary  of  Air  Force  Requirements  for 
1)  managing  and  developing  Air  Force  Systems.  2)  develop¬ 
ing  the  personnel  subsystems  of  Air  Force  Systems,  3)  testing 
Air  Force  Systems,  subsystems  and  equipments,  and  4)  test¬ 
ing  the  personnel  subsystems* 

Section  III  contains  an  overview  of  the  scientific  literature  as 
it  relates  to  the  assessment  of  human  performance,  and  sum¬ 
marizes  current  perspectives  on  analytical,  data  collection, 
and  evaluative  techniques. 

Section  IV  reviews  assessment  programs  and  techniques  in 
selected  Air  Force  systems,  most  of  which  information  was 
secured  from  SPC  and  contractor  personnel  and/or  test 
documents. 

Section  V  briefly  summarizes  and  concludes  the  report,  with 
emphasis  given  to  thj  implications  of  this  information  for  an 
overall  assessment  methodology. 
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Table  II.  Generalized  System  Engineering  and  Human  Performance  Assessment 
Activities  During  the  Development  of  the  Air  Force  System 
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REVIEW  OF  REQUIREMENTS  FOR  ASSESSING  HUMAN 
PERFORMANCE  IN  AIR  FORCE  SYSTEMS 


A*  Introduction 


Throughout  system  development,  numerous  decisions  are  made,  as  a 
matter  of  course,  concerning  the  functioning  and  structure  of  the  system 
The  process,  anticipated  in  the  foregoing  section,  may  be  thought  of  as  a 
series  of  successive  decisions  about  organizing  the  set  of  personnel-environ- 
ment-activity  interactions  called  the  system. 

In  order  *o  get  the  job  done  as  thoroughly  and  effectively  as  possible, 
system  development  is  guided  by  an  elaborate  set  of  government  directives 
and  requirements  documents.  These  documents  contain  the  policies  and 
needs  of  the  Air  Force  and  other  governmental  agencies  as  well  as  consen¬ 
sus  agreements,  between  the  defense  agencies  and  contractors,  concerning 
standards  of  good  management  and  engineering. 

This  section  overviews  and  summarizes  the  major  Air  Force  require¬ 
ment  documents  that  structure  (1)  the  management  and  development  of  the 
Air  Force  system;  (2)  the  development  of  a  personnel  subsystem  within  the 
over-vll  system;  (3)  the  testing  and  evaluation  of  the  system,  subsystems, 
and  various  equipments;  and  (4)  the  testing  and  evaluation  of  human  perform¬ 
ance  in  developing  systems.  Information  in  this  section  reflects  the  weapons 
system  development  policies  in  effect  during  the  1964-1965  period  and  those 
wh:ich  refer  more  specifically  to  human  performance  assessment.  This  in¬ 
formation  provides  the  context  within  which  evaluations  of  human  perform¬ 
ance  in  Air  Force  systems  typically  take  place. 

The  major  requirements  of  the  Air  Foice  in  the  above  areas  are  con¬ 
tained  in  different  types  of  Air  Force  and  military  documents  including  the 
following: 

Air  Force  Regulation  (AFR) 

Air  Force  Manual  (AFM) 

Air  Force  System  Command  Regulation  (AFSCR) 

Air  Force  System  Command  Manual  (AFSCM) 

Air  Force  System  Command  Program  Management 
Instruction  (AFSCPMI) 

Exhibits  originating  with  Divisions  of  Air  Force 
Systems  Command  (e,g. ,  BSD  Exhibits' 

Military  and  Federal  Standards 
Military  Specifications 
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The  conunt?  of  such  document*  are  represented  in  the  discussions  which  fol¬ 
low.  The  major  references  ere  tabled  later  in  this  section  and  are  keyed  to 
activities  of  particular  relevance  to  the  assessment  of  human  performance. 

A  representative  list  of  the  reference  requirtment  documents  is  included  in 
Appendix  71. 

B *  Requirements  for  Managing  and  Developing  Air  Force  Systems 

Various  documents  contain  the  policies  of  the  Air  Force  for  weapons  sys¬ 
tem  management  and  development.  The  principal  documents  are  Air  Force 
Regulations  375-1,  375-2,  375-3,  and  375-4.  An  excellent  overview  of  the 
Air  Force  Systems  Management  Concept  is  contained  in  AFR  375-4.  Of  par¬ 
ticular  interest  here  are  the  phases  within  which  Air  Force  Systems  are 
managed  and  developed.  Briefly,  there  are  four  phases:  Conceptual,  Defini¬ 
tion,  *  cquisition,  and  Operational. 

The  Conceptual  Phase  is  the  period  extending  from  the  determination  of 
a  broad  objective  until  the  Office  of  the  Secretary  of  Defense  (OSD)  approval 
of  the  Program  Change  Proposal  (PCP)  covering  the  Definition  Phase.  If 
the  Definition  Phase  does  not  apply,  the  Conceptual  Phase  extends  to  the  is¬ 
suance  of  the  System  Program  Directive  (SPD). 

In  the  Definition  Phase,  the  technological  advances  resulting  from  the 
Conceptual  Phase  are  translated  into  total  system  design  requirements  The 
SPO/Industry  team  utilizing  the  detailed  methods  and  procedures  described 
in  AFR  375-4  and  the  referenced  AFSCM  375-1  and  375-5  manuals  completely 
and  fully  investigates  all  aspects  of  existing  technology,  past  studies,  and 
possible  future  studies  to  define  the  requirements  outlined  in  the  SOR/OSR/ADO. 
The  basic  objective  of  this  effort  is  to  insure  that  full-scale  development  is 
not  started  until  costs,  schedules,  and  performance  objectives  have  oeen 
carefully  identified,  evaluated  against  one  another  and  a  high  probability  es¬ 
tablished  of  successful  completion  of  the  Acquisition  Phase. 

The  fundamental  objective  of  the  Acquisition  Phase  is  to  acquire  the  sys¬ 
tem  in  such  a  mariner  that  the  Specific  Opera  tional  Requirement  (SCR)  or 
Specified  Advanced  Development  Objective  (ADO)  is  met  while  at  the  same 
time  minimizing  system  cost,  time  to  acquire  the  system,  and  maximizing 
system  effectiveness.  The  orderly  transition  of  system  responsibility  and 
management  responsibility  from  AFSC  to  the  User  Coinmand  and  AFDC  is 
provided  for  during  the  overlap  of  the  Acquisition  and  Operational  Phases. 

The  Operational  Phase  has  been  established  as  a  logical  concluding  s*ep 
for  the  system  acquisition  cycle.  'rhe  prime  goal  of  this  phase  is.  successful 
system  operation.  Secondary  goals  are  adequate  system  support  by  the  AFLC 
and  an  orderly  turnover  from  AFSC  to  the  User. 

These  phases,  taken  together,  describe  the  life  cycle  of  the  Air 
Force  weapons  system.  More  or  less  involved  throughout  are  the  Air  Force 
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Headquarters  USAF,  Air  Force  Systems  Commend  (AFSC),  Air  Force  Lo¬ 
gistic*  Commend  (AFJLC),  Air  Training  Commend  (ATC),  end  the  User 
commends  (SAC,  TAC,  etc, ). 

Figure  3  summarise*  the  system  development  phases  and  major  activi¬ 
ties  or  products  of  the  functions  within  the  phases  of  system  life, 

C,  Requit ements  for  Developing  the  Personnel  Subsystems 

in  Air  Force  Systems 

The  requirements  to  develop  specific  products  and  elements  in  support 
of  the  over-all  man  ma:hine  performance  of  the  weapon  system  are  specified 
in  AFR  30-8,  The»e  requirements  are  listed  in  Appendix  III.  In  summary, 
the  Air  Force  requires  that  human  performance  be  developed  in  a  sy  item- 
atic,  timely  manner  coordinated  with  the  development  of  the  system?  hardware 
and  facilities.  This  usually  requires  specific  attention  to  human  prformance 
as  a  function  of  the  design  of  equipment,  safety,  layout  of  the  wpj  k  places; 
skills,  training,  number,  and  organization  of  the  system  personnel;  type, 
amount,  and  content  of  training  and  training  equipment;  and  operating  and 
maintenance  procedures.  The  systematic  development  of  these  system  elements 
requires  detailed  description  of  the  interactions  of  system  personnel  with  the 
system  environment  (including  equipment  and  ambient  conditions)  and  the  main¬ 
tenance  of  a  valid  and  reliable  centralized  body  of  basic  descriptive  informa¬ 
tion,  The  amount,  type,  and  format  of  this  information  may  vary  from  system 
to  system.  There  are,  however,  military  specifications  and  Air  Force  direc¬ 
tives  which  deal  directly  with  the  requirements  and  preparation  of  integrated 
man-machine  information  for  particular  systems.  These  have  been  noted  in 
the  summary  table  at  the  end  of  this  chapter.  The  primary  source  of  infor 
mation  and  guidance  for  developing  the  Personnel  Subsystem  is  AFSCM80-3, 
Handbook  of  Instructions  for  Aerospace  Personnel  Subsystem  Designers. 

D.  Requirements  for  Testing  Air  Force  Systems 

Air  Force  Regulation  80-14  is  the  controlling  requirements -document  for 
the  testing  and  evaluation  of  Air  Force  systems,  subsystems,  and  elements. 

Important  policy  explanations  and  distinctions  are  made  in  the  regulation 
concerning  the  types  of  tests  required  to  support  USAF  research  and  develop¬ 
ment  and  system  acquisition.  The  regulation  applies  to  all  Air  Force  organi¬ 
zations  and  activities.  “Testing"  refers  to  any  project  or  program  designed 


^The  discussion  of  these  documents,  relating  to  individual  systems  or  per¬ 
sonnel  subsystem  elements,  is  not  within  the  scope  of  this  chapter.  Our 
purpose  here  is  principally  to  point  out  and  describe,  as  briefly  as  possible, 
the  broad  context  of  performance  assessment. 
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Procurement  '  Development,  j 

and  Production^  Procurement _ f  Construction  Follow -on 

Management  Plan  and  Produc-  j  Productiot 

(Anp.  1-375-4)  tion  Contract  1 
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Product*  of  Functions  Within  the  Phases.  (Source:  AFSCM  375-4. ) 


to  obtain,  verify,  and  provide  data  for  evaluating  research  and  development, 
progress  in  accomplishment  uf  developmental  objectives,  and  performance 
and  operational  capabilit^of  systems,  subsystems,  components,  and  equip¬ 
ment  items  *  The  term  "evaluation"  i  s  construed  as  the  review  and  evaluation 
of  quantitative  or  qialitative  data.  Distinctions  are  made  concerning  types 
of  testing.  Research  testing  is  defined  in  4FR  80-14  as  a  project  designed 
to  verify  hypotheses  and  propose  solutions  to  operational  needs  or  accumu¬ 
late  new  knowledge.  Developmental  testing  is  an  integral  part  of  the  devel¬ 
opment  process  used  to  sense  trends,  measure  progress,  and  verify  the 
accomplishment  of  development  objectives  on  a  continuing  basis.  Anticipat¬ 
ing  t«e  discussion  next  of  human  performance  testing,  these  explanations 
provide  the  policy  basis  for  the  continuous  assessment  03  the  products  and 
processes  of  the  personnel  subsystem.  I 

AFR  80-14  specifies  the  oojectives,  policies,  and  procedures  of  testing 
and  delineates  three  functional  categories  of  formal  tests:  Category  I,  Cate¬ 
gory  II,  and  Category  III.  Catetory  I  and  II  testing  occur  during  the  Acqui¬ 
sition  Stage  of  system  development.  Category  IIJ  testing  is  carried  ^ut 
during  the  early  portion  of  the  Operational  Stage  of  the  system  life  cycle. 

Four  types  of  testing  are  also  described  in  the  document  and  are  subject  to 
the  three  functional  categories  or  phases  of  testing.  These  test  programs 
cover  the  testing  of  weapons  svstems,  research  and  development  projects, 
engineering  services,  and  ballistic  missiles. 

Excerpts  from  AFR  80-14  are  presented  in  AppendixIVand  in  Figure  4. 
Additional  instructions  ''oncerned  with  testing  and  evaluation  are  contained  in 
other  Air  Force  documents.  AFSC  Program  Management  Tnstruction  6-10  is 
particularly  useful  in  defining  and  providing  guidance  on  facv  'rs  common  to 
the  test  and  evaluation  of  weapons  systems  It  presents,  in  relatively  abbre¬ 
viated  form,  the  requirements  and  content  for  test  planning,  test  documen¬ 
tation,  programming  resources  and  test  items,  data  reduction  and  analysis, 
evaluation  of  test  results,  test  reporting,  and  the  classific  \tionof  procurable 
items  on  the  basis  of  favorable  test  and  evaluation  of  the  hems. 

E.  Requirements  for  Testing  and  Evaluating  the 

Personnel  Subsystems  in  Air  Force  Systems 

The  systematic  development  Oi.  the  human  performance  xn  Air  Force  sys¬ 
tems,  specified  in  AFR  30-8  and  gu  ded  by  other  doc ument s  (see  Table  IV  at 
the  end  of  this  section),  includes  te  ting  and  evaluation  in  accord  with  AFR 
80-14.  Formal  coordinated  testing  is  to  be  integrated  with  system  testing 
and  carried  out  from  Category  I  through  Category  III  until  it  is  verified  that 
the  system  can  be  operated,  maintained,  and  supported  by  USAF  personnel 
as  intended.  Formal  guidance  concerning  the  testing  and  evaluation  of  per¬ 
sonnel  subsystem  products  and  processes  that  support  the  man-machine 
performance  of  the  system  is  given  :in  AFSCM  30-3.  The  direction  is  un¬ 
equivocal  and  logical:  whenever  plans  are  made  to  introduce  a  human  function. 
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into  a  system,  plans  must  also  be  made  to  test  this  function  in  relarion  to  the 
rest  of  the  system.  .  .much  of  this  must  be  accomplished  in  early  stages  of 
system  design,  Wherever  feasible,  AFSCM  80-3  directs,  personnel  subsys¬ 
tem  test  data  should  be  collected  at  system  milestones  such  as  design  reviews, 
equipment  revisions,  demovistrations,  and  inspections .  These  may  include 
prototype  design  engineering  inspections,  mock-ups,  flight  demonstration  re¬ 
views,  qualification  testing,  etc.  In  addition,  failure  and  consumptive  data, 
reports  on  hazards,  accidents,  safety,  incidents,  etc.,  must  be  collected  and 
analyzed  for  human  performance  implications.  AFSCM  80-3,  following  AFP. 
80-14  and  other  documents,  alsodiscusses  the  Air  Force  functional  responsi¬ 
bilities  for  developing  and  testing  the  personnel  subsystem.  A  copy  of  the 
manual’s  summary  table  is  presented  in  Table  III. 

F .  Summary  of  the  Major  Requirements -Documents 

The  major  requii ements- documents  structuring  the  description,  classifi¬ 
cation,  prediction,  support,  and  verification  of  required  human  performance 
in  Air  Force  systems  are  presented  in  Table  IV.  The  documents  are  listed 
by  Air  Force  designation  and  classified  in  two  ways:  (1)  pertinence  to  the 
management,  development,  and  test  of  the  personnel  subsystem  and  allied 
topic  areas,  and  (2)  category  of  Air  Force  and  Department  of  Defense  docu¬ 
mentation.  This  information  illustrates  the  requirements  that  were  in  effect 
prior  to  or  during  the  period  of  this  study  (1964-1965). 
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Table  III.  Air  Force  Functional  Responsibilities  for  Personnel  Subsystem 
Development  (Source:  AFSCM  80-3,  page  A.  4-5) 


***Primary  Responsibility  ^Participation  ^Coordination 


Ill 


OVERVIEW  OF  THE  TECHNOLOGY  OF  THE  BEHAVIORAL 
SCIENCES  FOR  ASSESSING  HUMAN  PERFORMANCE 


A.  Introduction 


This  section  is  written  principally  as  a  point  of  departure  for  an 
understanding  of  the  technology  of  the  behavioral  sciences  for  assessing 
human  performance  in  military,  particularly  Air  Force,  systems.  It 
contains  (1)  our  perspectives  on  the  broad  behavioral  science  technology 
applicable  to  studies  of  human  performance  in  the  system  context  and  a 
schema  useful  for  mapping  the  relevant  experiences  and  literature  of  the 
behavioral  sciences;  (2)  a  review  of  selected  areas  of  the  behavioral 
science  technology  relevant  to  the  description  and  evaluation  of  human 
performance  ?n  the  military  system  context. 

B.  Relevant  Areas  of  Research  and  Technology  in  the  Behavioral 
Sciences 


There  are  five  areas  of  information  and  methodology  in  the  behavioral 
sciences  which  are  particularly  relevant  to  the  assessment  of  human  per- 
formance  in  Air  Force  systems.  We  derive  these  five  areas  or  foci  by 
elaborating  on  the  perspective,  presented  in  Section  I,  that  studying  human 
performance  involves  focussing  on  some  system  PERSONNEL,  in  the 
system  ENVIRONMENT  (which  includes  other  system  personnel,  system 
equipment  and  the  ambient  system  conditions),  carrying  out  some  selected 
ACTIVITY  or  set  of  activities  in  exe  ution  and/or  support  of  the  system 
mission.  The  five  types  or  aspects  of  performance  are  described  br  .tfly 
here. 


The  first  ’•type”  has  to  do  with  the  functioning  of  the  individual 
operator,  or  maintenance  technician,  etc.  ,  Vvith  his  capacities  and  limitations. 
More  specifically,  this  means  the  individuals  sensing,  sensory  organization, 
responding,  drive/ mo  tivatiomil,  storage  /memory,  and  feedback/adaptive 
functioning  (see  Figure  1  in  Section  I  for  a  basic  model  of  individual  human 
functioning).  The  second  aspect  of  human  performance,  conceived  in 
personnel-environment-activity  terms,  is  the  inter-personnel  inter action(s). 
Study  of  interpersonal  performance  involves,  for  example,  consideration  cf 
social  structure,  communication  networks,  authority  arrangements,  roles  and. 
group  syntality.  The  third  ■•type*1  of  performance,  widely  studied  ia  the  typical 
human  factors  or  human  engineering  program,  is  the  interaction  of  system 
personnel  and  the  various  system  equipments.  Studies  of  these  interactions 
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generally  focus  upon  levels  of  automation,  man-machine  dynamics,  control- 
display- behavior  relationships,  etc.  The  fourth  "type"  of  performance  sug¬ 
gested  by  the  personnel- environment- activity  dimensions  of  human  perform¬ 
ance  is  the  interaction  of  system  personnel  and  the  ambient  system  conditions, 
for  example,  illumination,  sound,  air  conditioning,  sensory  and  social  isola¬ 
tion.  And  the  fifth  interaction  of  relevance  in  the  assessment  of  human  per¬ 
formance  is  between  system  personnel  and  the  system  activities,  procedures, 
and  doctrine. 


The  interactive  per sonnel-environment- activity  concept  of  human  per¬ 
formance,  in  addition  to  making  for  a  surrmary  of  the  subject  matter  of  the 
applicable  behavioral  science  technology,  also  helps  identify  the  behavioral 
science  specialists,  disciplines  and  published  literature. 


Personnel  specialists  and  thc.se  interested  in  the  impact  of  individual 
human  differences  upon  system  effectiveness  typically  do  or  can  attend  to 
the  performance,  capacities,  and  limitations  of  individual  system  personnel. 
They  can  be  assisted  bv  the  familiar  QQPRI,  the  training,  and  the  proficiency 
evaluation  specialists.^ 


Social  psychologists,  group  dynamicists,  unit  proficiency  analysts, 
cultural  anthropologists,  game  theorists,  and  such  specialists  can  and  often 
do  contribute  to  the  analysis  and  understanding  of  the  inter-personnel  inter- 
actions.  ^ 


Human  factors  engineering,  industrial  design,  and  such  disciplines 
typically  are  involved  with  the  per sonnel- equipment  interactions.  °  Life 
support  scientists,  safety  engineers,  and  physiologists  are  among  the  pro¬ 
fessionals  whose  technology  is  applicable  to  the  interaction  of  system  per¬ 
sonnel  and  the  ambient  system  conditions.  ( 


4 

see,  for  example,  references  2,  21,  73,  78,  79,  84,  88,  91,  97,  99,  105, 
107,  108,  111,  112,  124,  133,  135,  138,  141,  143,  150,  157,  160,  171,  172, 
183,  184,  332,  337,  359,  363,  364,  419. 

5 

see,  for  example,  references  6,  12,  42,  59,  77,  81,  82,  103.  110,  125. 

152,  163,  289,  324,  376,  401. 

^see,  for  example,  references  40,  51,  52,  63,  66,  80,  92,  98,  99,  H9,  131, 
168,  171,  172,  185,  186,  312,  314,  406,  444,  446. 

7 

see,  for  example,  references  94,  140,  152,  177,  198,  341,  374,  393,  438, 
445. 
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Work  methods  and  industrial  engineers,  training  professionals,  and 
handbook  specialists  are  involved  typically  the  development  of  the  desired 
personnel-activity  interactions  in  the  operation  and  maintenance  of  a  military 
system,  8 

As  Air  Force  systems  develop  through  the  conceptual,  definition* acqui¬ 
sition,  and  operational  phases  of  their  life  cycles,  a  number  of  evaluative 
activities  take  place  with  reference  to  the  interactions  among  system  person¬ 
nel  and  the  system  environment,  and  procedures,  activities  and/or  doctrine 
(see  Table  II  in  Section  I  for  a  summary  of  system  and  human  performance 
studies).  In  general,  these  activities  (1)  analyze  human  performance  in  terms 
of  system  requirements,  (2)  collect  empirical  information  to  enable  prediction 
and/or  verification  of  system  /human  performance,  and  (3)  evaluate  and  use 
such  information.  This  assessment  process  is  supported  by  a  repertoire  of 
behavioral  science  studies  principally  to  be  found  in  four  major  fields  (3). 

The  fields  are  (1)  human  performance,  (2)  personnel  research,  (3)  human 
support  and  maintenance,  and  (4)  economic  analysis  and  management. 

Figure  5  schematizes  the  preceding  discussions  and  summarizes  the 
approach  which  we  found  useful  in  the  methodological  review  of  the  behavioral 
science  literature  for  assessing  human  performance  in  Air  Force  systems. 
Figure  6  relates  the  schema  of  this  methodological  review  to  the  personnel- 
environment-activity  concept  of  human  performance.  We  regard  the  develop¬ 
ment  and  reporting  of  this  mapping  or  plan  for  searching/ reviewing  the 
relatively  vast  behavioral  science  literature  as  important  to  this  overview 
and  as  potentially  useful  to  those  involved  in  the  assessment  of  human 
performance. 

C.  Major  Technologies  of  the  Behavioral  Sciences  Relevant  to 

Human  Performance  Assessment  in  Air  Force  Systems 


The  foregoing  discussion  indicates  the  broad  content  and  methodological 
areas  or  fields  of  the  behavioral  sciences  relevant  to  the  assessment  of 
human  performance  in  military  systems.  The  development  of  the  behavioral 
sciences  to  their  present  day  involvements-in  system  development  is 
generally  important  and  interesting,  but  is  not  discussed  here  (see,  for 
example,  references  13,  21,  48,  96,  142,  382,  and  385).  Rather  we 


®see,  for  example,  references  5,  8,  60,  69,  78,  83.  8  5,  93,  97,  99,  105, 
120,  122,  123,  128,  139,  147,  148,  158,  159,  163,  165,  166,  174,  176, 

305,  428,  442. 
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Figure  5,  Approach  to  the  Methodological  Review  of  the  Behavioral  Sciencef 
for  Assessing  Human  Performance  in  Air  Force  Systems 
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Figure  6.  Refined  Mapping  of  the  Methodological  Studies,  Perspectives*, 
and  Instruments  ox  the  Behavioral  Sciences  Reviewed  During 
This  Study 
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wish  to  draw  attention  to  what  wo  believe  to  bo  the  two  technologic*  within 
the  behavioral  sciences  moat  useful  to  the  study  of  human  performance  in 
the  context  of  the  military  and  industrial  system*  These  technologies 
include  (1)  practices  for  conceptually  assessing  human  performance,  and 
(2)  practices  for  empirically  assessing  human  performance.  By  the 
conceptual  assessment  of  human  performance,  we  mean  the  description, 
understanding,  and  evaluation  of  human  functioning  prior  to  the  actual 
manning,  by  representative  personnel,  of  the  system  roles  and  positions. 
Conceptual  assessment  usually  takes  place  during  the  early  system  life 
stages  when,  largely,  the  projected  system  personnel,  system  equipment, 
and  system  environment  are  unavailable,  and/or  not  yet  developed  or 
obtained.  Empirical  assessment,  on  the  other  hand,  refers  to  the  collect!  >n 
and  analysis  of  information  concerning  observable  system  personnel  in  the 
real  or  simulated  system  environment  carrying  out  some  system  activity  or 
procedure.  Typically,  empirical  assessment  is  carried  out  at  system 
simulation  facilities,  prototype  test  facilities,  and  operational  system  sites. 
Empirical  assessment  activities  generally  are  not  possible  or  opportune  dur¬ 
ing  early  system  development  stages,  e.g.,  conception,  definition.  T>  ey  take 
place  usually  during  the  acquisition  and  operational  stages  of  system  li 

1 .  Techniques  for  the  Conceptual  Assessment*1  of  Human 
Performance 


The  study  of  human  performance  during  the  early  system  life 
stages  requires  the  modeling  and  systematic  gaming  of  the  expected  role(s) 
and  functioning  of  the  human  component  in  the  system.  Largely,  this  ex¬ 
amination  of  human  performance  entails  conceptual  analyses,  paper  and 
pencil  studies  both  quantitative  (mathematical)  and  non- quantitative  (verbal, 
diagrammatic,  etc,). 


Historically,  behavioral  scientists  have  not  frequently  used  mathe¬ 
matical  models  in  their  study  of  human  performance.  This  would  appear  true 
not  so  much  because  the  subject  matter  is  not  amenable  to  quantitative  analyses 
and  expression  but,  probably,  because  the  practitioners  of  the  behavioral 
sciences  have  not  been  accustomed  to  think  and  communicate  in  mathem? rleal 
terms.  This  is  not  surprising:  mathematics  has  developed  as  the  hand¬ 
maiden  of  the  physical  sciences.  On  the  other  hand  the  behavioral  scxent:st 
has  used  narrative  and  diagrammatic,  (eg.,  flow  diagrams),  techniques  to 
communicate  about  tie  subject  matter  of  human  performance.  It  should  be 
pointed  out,  however,  that  these,  as  with  mathematical  models,  are  model¬ 
ing  techniques,  committing  to  paper  symbols  and  relations  among  symbols 
that,  in  effect,  stand  for  some  aspect  of  performance.  Currently,  in  the 
behavioral  sciences,  a  variety  of  mathematical  and  verbal  modeling  tech¬ 
niques  is  used  in  the  study  of  human  performance  and  shoild  be  noted  here 
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regarding  the  study  of  the  "conceptual  man"  at  the  early  stages  of  system 
life,  _ 


a .  Quantitative  Heuristic  Techniques 

The  extension  of  classical  mathematical  analysis,  concerned 
with  a  few  variables  and  their  continuous  functional  interrelationships,  to 
the  behavioral  sciences  has  been  marked  with  some  success,  e.g, ,  he  wide¬ 
spread  'ise  of  classical,  quantitative  correlational  techniques,  and  with  certain 
problems.  The  first  of  these  problems  concerns  the  traditional  necessity 
in  classical  mathematics  to  Isolate  a  relatively  few  variables.  The  study 
of  a  lew  variables  works  well  m  carefully  contrived  laboratory  settings  but 
is  patently  difficult  in  the  typical  man-machine  system  study.  The  second 
weakness,  (and  the  principal  one  for  this  review),  of  classical  mathematics 
is  that  it  has  not  successfully  coped  with  the  problem  of  determining  the 
important  variables:  the  classical  method  works  best  when  it  can  deal  with 
variables  that  are  given,  known  in  advance.  And,  yet  to  he  useful  in  early 
system  design,  modeling  techniques  must  allow  the  discovery  or  recognition 
of  aspects  of  human  functioning  to  which  specific  attention  should  then  be 
given.  In  brief,  the  import  of  classical  mathematical  techniques  in  the 
*  striving  of  the  behavioral  sciences  to  under  b\v  and  plan  for  humans  in  sys¬ 
tem  context  is  that  they  have  been  and  conti*  to  be  powerful  tools  for  de¬ 
ducing  relations  among  performance  variable  but  they  are  ».ot  particularly 
useful  for  discovering  performance  variables. 

More  recent  mathematical  techniques  have  been  developed 
which  can  contribute  to  discoveries  and  hypothetical  positions  regarding  human 
performance.  Fox  the  most  part,  these  techniques  are  heuristic:  they  pro¬ 
vide  concepts  with  which  the  system  designer  can  work,  with  which  he  can 
describe,  classify  and  deduce  properties  of  the  man-environment-activity 
interaction.  Examples  of  such  approaches  are  topological  theory,  graph 
theory,  and  the  theory  of  games.  These  theoretics  represent  departures 
from  the  classical  type  of  thinking  symbolized  by  the  expression  x  *  f(y). 
Instead,  they  focus  upon  descriptions  of  organized  systems  and  provide  the 
anatomy  of  the  human  performance  cf  interest,  The3e  approaches  help  identify 
performance  variables  which  can  then  be  studied  by  ordinary  mathematics. 

For  example,  the  network  of  interpersonal  relations  in  an  operating  crew  or 
military  organization  can  be  schematized  by  a  linear  graph  (in  the  graoh 
theoretic  approach and  the  social  properties  involved  in  the  group's 


^See  Harary  and  Norman  (126)  for  an  examination  of  the  potential  use  of 
graph  theory  in  the  behavioral  sciences. 
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performance  can  be  expressed  as  mathematical  properties  of  the  graph* 

The  usefulness  of  doing  this  is  that  heretofore  unsuspected  variables  or 
characteristics  of  them  may  be  opened  up  for  study  and  that  the  system 
designer  is  provided  with  suggestions  as  to  how  to  proceed  further. 

b.  Non* Quantitative  Heuristic  Techniques 

The  foregoing  paragraphs  point  up  the  usefulness  of  quanti¬ 
tative  functional  modeling,  us  mg  the  newer  mathematical  approaches,  in 
the  conceptual  assessment  of  human  performance.  It  should  be  noted,  how¬ 
ever,  that  most  j  /stem  and  human  performa  nce  analyses  start  and  continue 
with  non- quantitative,  principally  verbal,  descriptions.  This  section  contains 
a  tabulation  of  an  illustrative  group  of  techniques  which  have  been  developed 
and  found  useful  for  examining  qualitatively  (verbally,  diagrammatical!  y, 
etc.  )  the  interactions  among  system  personnel,  system  environment  and 
system  activities. 

Current  practices  in  the  non- quantitative,  heuristic  examina¬ 
tion  of  human  performance  seem  to  be  the  result  of  two  lines  of  development 
in  the  behavioral  sciences:  (1)  the  description, analysis,  and  classification  of 
human  work,  and  (2)  th^  rating,  evaluation,  and  comparison  of  incumbents 
and  jobs.  Work  description  and  analysis  includes, historically, numerous 
approaches  toward  operationally  defining  and  structuring  the  content  of 
human  work  (eg.,  job  description  and  job  analysis),  ^  the  process,  work¬ 
place  and  movements  of  v/ork  (e.  g.  ,  time  and  motion  analysis),  ^  and  the 
hierarchies  of  jobs  and  incumbents  (e.  g.  ,  occupational  analysis  and  classifi¬ 
cation).  13  Of  particular  note  for  this  review  is  the  post- World  War  II  applica¬ 
tion  of  work-analytic  techniques  to  the  planning  for  and  design  of  Air  Force 
systems l^and  the  comparatively  recent  development  of  the  U.S.A.F.  Personnel 


Useful  discussions  of  mathematical-psychological  methodology  are 
contained,  for  example,  in  Rapoport  (163)  and  Luce  et  al.  (141). 

**see,  for  example,  references  10,  90,  93,  94,  98,  99,  100,  101, 
105,  113,  120,  137,  147,  151,  153,  158,  164,  174,  181,  381,  383, 
442. 

12 

see,  for  example,  references  83,  120,  123,  153. 

13 

se*\  for  example,  references  142,  165,  183,  265,  278. 

1  x 

see,  for  example,  reference  151. 
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Subsystem  approach  based  in  largo  part  upon  the  construction  and  use  of 
function-  and  task-analytic  information*  15 

Table  V  presents  and  summarises  the  major  non-quantitatlv* 
techniques  useful  for  examining  human  performance  heuristic  ally.  The 
contents  of  the  techniques  are  ascribed  Mong  the  dimenflions  of  the  per 
sonnel  environment-activity  concept  of  human  performance.  The  tech¬ 
niques  have  been  classified  according  to  format:  giaphic,  tabular, 
narrative,  and  lists  and  rating  scales.  The  graphic  group  includes 
diagrams,  charts,  profiles  and  matrices.  The  narrative  and  tabular 
categories  are  self-explanatory;  the  lists  and  rating  scales  group  includes 
checklists  some  of  which  are  accompanied  by  rating  scales. 

Referring  to  the  table,  the  items  identified  under  each  of  the  gen¬ 
eral  categories  of  system  activity,  system  personnel,  system  equipment 
and  environment  should  oxpla  n  themseKes  Howeve  %  some  explanation  is 
required  for  the  two -level  category  of  activity.  Actually,  four  levels  can 
be  inferred  by  combining  the  information  in  the  "lever*  and  "relationship" 
categories,  H  the  function  level  is  checked  and  the  relationships  shown  are 
"within,"  then  the  level  of  analysis  handled  by  the  technique  concerns  sub¬ 
functions,  If  the  "task"  level  is  checked  along  with  the  "within”  relation¬ 
ship,  it  can  be  inferred  that  the  technique  analyzes  subtasks  or  elements. 
Additional  clarifications  related  to  items  within  a  specific  technique  are 
contained  in  the  footnotes  accompanying  the  table. 

The  content  analysis  inoicates  that  there  are  considerable  over¬ 
laps  among  techniques  and  that  techniques  differ  very  often  in  appellation 
rather  than  in  content.  For  example,  task  analysis  techniques,  such  as  the 
Operational  Sequence  Diagram,  Information-Decision-Action  Chart,  Tusk- 
Equipment  Analysis,  and  Position  Descriptions  are  all  very  similar. 
Differences  among  techniques  are  due  principally  to  the  format,  the  pur¬ 
poses  and  interests  of  the  analyst,  and  the  applu  .non  of  the  technique  at 
different  stages  of  system  development. 

Environmental  information  is,  in  general,  not  included  in  approx¬ 
imately  half  of  the  techniques  illustrated  in  this  review.  For  some  tech¬ 
niques,  perhaps,  the  information  is  not  pertinent  to  the  analysis  or  it  is 
analyzed  elsewise.  It  is  surprising,  however,  that  human  task  analytic 
techniques  do  not  contain  such  information  as  a  matter  of  rule. 


*^see,  for  example,  references  4,  13,  1  9,  20,  21,  22,  23,  29,  34,  37,  38, 
39,  45,  46,  48,  56,  57,  62,  63,  66,  08,  70,  71,  "  5,  86,  95,  v6,  114, 

116,  130,  131,  138,  151,  167,  173,  199.' 
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•able  V.  Content  Analysis  of  Typical  Non- Quantitative  Heuristic  Techniques 
Used  to  Examine  Human  Performance  in  System  Operation 
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References  for  each  technique  are  contained  in  Table  VI. 

Does  nol.  appear  on  all  time-lir.o  charts. 

Accompanied  with  a  time-line  chart. 

Utilize  a  numor  or  symbol  code  ascribable  to  a  position  title  or  equipment  name, 


Duties  and  Tasks  are  described  in  a  tabular  format. 
Also  includes  data  in  narrative  format. 


Summary  descriptions  of  these  techniques  are  contained  in  Table 
VI,  In  addition  to  the  description,  the  typical  uses  of  the  information  and 
the  major  proponents  of  the  technique  are  given.  The  list  of  proponents 
and  references  is  not  exhaustive  and  is  intended  principally  to  convey  the 
general  direction  of  the  literature  with  respect  to  verbal-heuristic  analyses 
of  human  performance. 

2.  Techniques  for  the  1  Empirical  Assessment11  of  Human 
Performance 


Thus  far,  this  section  has  summarized  the  technology  of  the  behav¬ 
ioral  sciences  for  examining  human  performance  while  the  performance  is 
largely  conceptual.  The  emp'o-us  which  follows  is  upon  practices  for 
examining  human  performance  empirically,  that  is,  for  planning,  conducting, 
and  using  a  test  situation  for  assessment  purposes.  System  personnel  in  the 
system  environment  carrying  out  some  system  activity  are  in  focus  during 
the  empirical  test  as  they  are  in  the  previously  described  paper-and-pencil 
analyses  of  human  performance. 

There  are  many  good  reference  workb  on  scientific  research  and 
the  conduct  of  behavioral  science  studies.  Some  are  philosophical  treatises 
on  the  scientific  method  (for  example,  14);  some  discuss  the  practice  of  the 
scientific  method  (for  example,  197);  many  are  compendia  of  methods  and 
findings.  ^  Overviewing  these  materials  here  is  difficult;  presenting  what 
they  have  to  say  in  great  detail  is  not  intended.  In  general  trrms,  however, 
they  advise  on  and  illustrate  principally  three  areas  of  the  technology  of  the; 
behavioral  sciences  for  empirically  assessing  human  performance.  These 
areas  are  (a)  planning  the  test  situation,  (b)  collecting  information  concern¬ 
ing  human  performance,  and  (c)  reducing,  analyzing,  and  using  the  obtained 
information.  17 

a.  Planning  the  Test  Situation 

Planning  the  test  situation  usually  involves  the  choice  and 
statement  of  a  researchable  or  testable  objective,  searching  and  reviewing 
relevant  literature  (as  available),  understanding  the  assessment  process  in 


^for  example,  118,  121,  142,  155,  179,  190,  199  (experimental  psychology); 
6,  12,  103,  254  (social  psychology);  4,  13,  16,  19,  20,  21,  22,  23,  29,  34, 
37,  38,  39,  45,  46,  48,  56,  62,  63,  66,  68,  71,  75,  86,  95,  96,  114,  116, 
130,  131,  138,  151,  167  (human  factors  engineering) 

17 

Wilson  (197)  indicates  that  ‘'many  scientists  owe  their  greatness  not  to 
their  skill  in  solving  problems  but  to  their  widsom  in  choosing  them.  !l 
The  many  guidelines  and  references  on  designing  tests  and  studies  not¬ 
withstanding,  there  seems  to  be  little  available  advice  and  few  specific 
techniques  for  knowing  when  or  that  a  test  situation  is  required. 
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Table  VI.  Summary  Descriptions,  Reported  Uses  and  Principal  Proponent(s)  of  Major 
Non- Quantitative  Techniques  to  Describe  Human/Sy stern  Performance 
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Table  VI*  Summary  Descriptions,  Reported  Uses  and  Principal  Proponent(s)  of  Major 
Non -Quantitative  Techniques  to  Describe  Human/System  Performance  (cont 
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Table  VI.  Summary  Descriptions,  Reported  Uses  and  Principal  Proponent(s)  of  Major 

Non- Quantitative  Techniques  to  Describe  Human/ System  Performance  (continued) 
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Table  VI.  Summary  Descriptions,  Reported  Uscjs  and  Principal  Proponent(s)  of  Major 

Nop- Quantitative  Techniques  to  Describe  Human/System  Performance  (continued) 
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Table  VI.  Summary  Descriptions,  Reported  Uses  and  Principal  Proponent  (»)  of  Major 

Non- Quantitative  Techniques  to  Describe  Human/System  Performance  (continued) 
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Table  VI.  Summary  Descriptions,  Reported  l^ses  and  Principal  Proponent^*)  of  Major 
Non -Quantitative  Techniques  to  Describe  Human/Svstem  Performance  (con* 
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Table  VL  Summary  Descriptions,  Reported  Uses  and  Principal  Proponent(s)  of  Major 
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Table  VI.  Summary  Descriptions,  Reported  Uses  and  Principal  Froponent(s)  of  Major 

Non- Quantitative  Techniques  to  Describe  Human/System  Performance  (continued) 
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terns  ©f  the  seientifie  method  (for  ©xau possessing  a  commitment  to 
ligorious  and  yet  parsimonious  study  of  -  i©  problem,  conducting  the  study 
as  ©lamed,  and  controlling  the  quality  and  interpretation  of  the  information), 
and  the  careful  design  ©f  th*  test  situat  an  (so  as  to  identify  clearly  the 
variables  to  be  measured  and  those  to  be  manipulated  or  otherwise  con¬ 
trolled  and  the  conditions,  equipment,  and  procedures  of  the  test),  *8 
Planning  of  studies  of  human  performance  in  systems  contexts  requires 
also  the  careful  analyses  and  explicit  statement  of  the  relationship  between 
the  human  performance  of  interest  and  the  system  perfoimance  so  as  to 
justify  the  selection  and  execution  of  the  study, .  to  communicate  with  other 
specialists  engaged  in  the  design,  development,  and/or  the  operation  of  the 
system,  and, ultimately,  to  benefit  and/or  assure  demonstrably  the  desired 
system  performance,  In  addition,  designing  ar.d/or  taking  best  ad¬ 

vantage  of  test  situations  during  the  development  of  the  system  implies  a 
thorough  understanding  of  the  system  development  process  arid  the  needs  for 
timely,  sometimes  cruci?l,  information  concerning  human  performance,^0 

b.  Techniques  for  Collecting  Information 

The  behavioral  literature  eentains  many  references  to  and 
techniques  for  the  collection  of  data  in  empirical  situations.  The  techniques 
useful  in  the  empirical  situation  are  variations  of  the  basic  method  of  science: 
controlled  observation.  Observation  may  be  direct,  as  when  the  human  ob¬ 
server  is  physically  and  temporally  at  the  test  locale,  or  indirect.  The  in¬ 
direct  observer  often  uses  interviews  to  establish  information.  He  can  also 
read  reports.  Observation  may  involve  the  participation  of  the  experimenter, 
test  administrator,  or  test  specialist^ )  or  it  may  not. 


18 

The  design  and  planning  of  tests  and  studies  of  human  performance  is 
discussed  in  13,  95,  96,  109,  118,  121,  124,  127,  142,  158,  160, 
179,  183,  189.  190,  192,  197,  287,  288,  292,  293,  296,  298,  304, 

384, 

19 

Relating  human  performance  to  system  effectiveness  is  addressed 
variously  in  2,  9,  28,  32,  35,  36,  65,  136,  145,  194,  195,  315, 

320,  394,  403,  431,  442. 


20*ee  18,  23,  29,  30,  3 2,  41,  47,  49,  53,  54,  58,  64,  65,  70,  76,  130, 
131,  164,  170,  359,  394,  395,  404,  405,  408,  410,  411,  412,  416, 
418,  420,  422,  423,  427,  443. 
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The  behavioral  science  literature  concerned  with  the  collection 
el  human  performance  information  contains  techniques  lor  instrumenting, 
structuring,  or  formatting  the  available  information*  Techniques,  such  as 
checklists  and  rating  scales,  are  designed  to  structure  both  the  acquisition 
and  the  deposition  of  relevant  information  and  to  facilitate  th^  tr  sk  of 
information  processing  beyond  the  collection  stage,  V'here  criterion  informa¬ 
tion  is  available,  these  instruments  serve  also  as  evaluative  instruments 
providing  for  the  selection  and  recording  of  valued-drtta  paints  by  the  ©baurver/ 
evaluator,  A  large  body  of  literature  has  been  developed  on  the  use  of  struc¬ 
tured  observation  forms  and  the  design  of  such  instruments  for  U\e  best 
possible  validity  and  reliability,  table  VII  contains  descriptions  of  the 
principal  techniques  appearing  in  the  behavioral  science  literature  and  useful 
for  collecting  information  in  the  empirical  situation. 

Meat  of  the  direct  observational  techniques  (e  £, ,  interviews, 
rating  scales,  checklists,  critical  incidents,  time  and  m  .Uon  practices,  per¬ 
formance  tests)  have  long  histories  of  u^e  in  the  description  and  analysis  of 
human  behavior.  Several  of  the  indirect  techniques  also  have  been  used  exten¬ 
sively  in  the  measurement  of  human  behavior,  both  in  the  laboratory  setting 
(e,  g, ,  response  timers,  pen  recorders,  questionnaires),  and  in  the  more 
applied  field  settings  (e,  g, ,  paper-pencil  surveys,  rating  scales,  questionnaires, 
paper-pencil  tests).  There  are  signs  that  some  newer  practices,  such  as 
video  recording,  the  use  of  CRT  remote  displays,  and  the  use  of  computers  to 
record  and  analyse  data  on  a  real-time  basis  are  and  will  b  come  increasingly 
useful.  It  is  not  uncommon  to  encounter  discussions  concerning  automated 
laboratories  in  which  the  stimulus  program,  data  recording,  and  data  analysis 
are  all  handled  by  a  computer.  This  study  reviewed  several  such  laboratories. 
All  that  is  required  of  the  human  subject  is  that  he  appear  and  follow  the  instruc¬ 
tions  generated  by  the  computer.  The  experimenter  need  not  even  be  present, 
although  he  often  is.  There  definitely  appears  to  be  a  trend  toward  some  form 
of  automation  in  experimental  laboratory  settings  and,  in  fact,  even  in  field 
settings,  e,  g, ,  the  integrated  test  course  for  the  measurement  of  combat 
effectiveness  being  built  for  the  U.  S.  Army  Quartermaster  Corps,  This  out¬ 
door  course,  which  is  capable  of  aU- weather  operation,  will  measure  human 
performance  via  remote  automatic  sensors  which  are  linked  to  a  central 
computer  which  records  and  analyzes  subject  responses  (309), 

c.  Techniques  for  Reducing  and  Analyzing  Information 

Data  reduction  usually  follows  data  collection  and  is  common¬ 
place  In  empirical  assessments  of  human  performance.  The  reduction  step  is 
one  of  summarizing,  making  the  information  manageable  and  amenable  to 
additional  analyses,.  The  result  of  this  step  is  typically  a  narrative  sum¬ 
mary  or  a  numerical  expression  summarizing  tie  counting  of  events  or 
other  measures  of  human  performance.  Quantified  summaries  usually  take 
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Table  VII.  Summary  Description  of  Major  Quantitative  Techniques  for  Gathering 
Empirical  Information  Concerning  Human  Performance  (continued) 
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statistical  ferns  anti  refer  to  the  trends  or  tendencies  in  the  information 
anti/or  variations  within  it>  ^  Common  statistical  forms  are  measures  o* 
central  tendency  (such  as  the  mean  median  mode),  measures  of  variability 
(average  and  standard  deviation,  range  of  variation,  etc. ),  percentages, 
proportions,  percentiles  and  frequency  distributions. 

The  behavioral  literature  contains  much  discussion  on  the  related 
topics  of  validity  and  reliability  of  observations  and  measures  concerning 
human  performance.  Validity  refers  to  the  utility  of  a  measure  or  obser¬ 
vation  for  some  purpose;  the  degree  to  which  the  observation  corresponds 
with  the  need  for  it.  Behavioral  studies  generally  illustrate  four  types  of 
validity;  content,  concurrent,  predictive,  and  construct.  Content  validity 
is  a  logical  validity  based  upon  the  opinions  of  qualified  people  that  measures 
of  human  performance  adequately  sample  the  performance  on  which  informa¬ 
tion  is  required  for  evaluation.  Concurrent  and  predictive  validities  are 
statistical;  the  degree  of  agreement  of  the  measure  and  its  purpose  are 
determined  by  correlational  procedures.  Construct  validity  is  a  logical 
validity  in  which  estimates  arc  made  of  the  degree  to  which  a  measure  or 
observation  of  human  performance  reflects  some  underlying  human  ability 
or  function.  The  concept  of  the  reliability  of  measurement,  differentiated 
from  the  reliability  of  equipment  operation  (probability  of  operation  at  a 
given  time)  is  estimated  by  the  statistical  agreement  among  repeated  obser¬ 
vations  and  there  are  a  number  of  statistical  expressions  referring  to  the 
absolute  and  relative  reliability  characteristics  of  the  measures. 

Reliability  and  validity  are  related  concepts  in  the  striving  to 
obtain  useful  information  concerning  human  performance.  Increasing  the 
reliability  of  the  information  often  involves  acquiring  measures  more  closely 
resembling  each  other  and,  therefore,  possibly  more  close  to  the  "truth11 
about  human  performance.  However,  a  set  cf  observations  might  be  highly 
consiitent  but  not  true.  On  the  other  hand,  highly  valid  or  useful  measures 
necessarily  involve  high  reliability  or  consistency,  A  number  of  excellent 
references  for  both  concepts,  reliability  and  validity,  are  to  be  found  in  the 
behavioral  literature,  e.  g, ,  Lindquist  (319). 

Various  statistical  techniques  have  been  developed  for  deriving 
meaning  from  the  summarized  observations  of  human  performance.  They 
generally  fall  into  two  types,  parametric  and  nonparametric,  depending  upon 
the  assumptions  which  can  be  made  concerning  the  types  of  measures  and 
certain  of  their  statistical  characteristics.  Several  techniques  for  determining 
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References  for  this  section  include, for  example,  288,  294,  297,  298,  300, 
301,  302,  303,  306,  307,  310,  311,  313,  317,  318,  319,  323,  329,  328,  331, 
333,  335,  338,  339,  340,  350,  355,  366,  370,  373,  385,  386,  387,  390. 
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degree  of  association  and  the  statistical  significance  of  differences 
between  sets  of  information,  e.  g.  ,  expected  human  performance  and 
observed  performance,  have  been  developed  and  are  currently  in  use. 

They  are  described  briefly  in  Table  VIII, 

D.  Summary  and  Overview 

The  phenomena  of  human  performances  occur  without  benefit  of  system 
designer  or  behavioral  scientist,  but  the  data  of  man-machine  performance 
are  the  joint  product  of  analysts  and  observers  and  the  phenomena- -coupled 
with  specially  contrived  data -generating  situations  and  techniques.  This 
chapter  has  reviewed  the  repertoire  of  the  behavioral  sciences  for  generating 
data  from  available  information^  and  for  deriving  meaning  from  them. 

For  the  most  part,  what  was  seen  in  the  state  of  the  art  as  a  result  of 
this  review  was  a  variety  of  familiar  techniques  and  approaches  in  the 
assessment  of  human  performance.  The  principal  methods  for  deriving 
information  concerning  human  performance  are  variations  of  logical  assump¬ 
tion,  indirect  human  observation  (e,  g, ,  interviewing),  and  direct  human 
observation,  The  principal  methods  for  deriving  meaning  from  the  informa¬ 
tion  rely  upon  human  reasoning  and  involve  comparison  with  previous  infor¬ 
mation.  Placing  a  value  upon  the  discovered  facts  requires  judgment  by  and, 
often,  consensus  among  the  users  of  the  evaluated  information. 

With  reference  to  the  assessment  process,  the  literature  of  the 
behavioral  sciences  contains  numerous  methodological  studies,  guidelines 
and  handbooks  which  provide:  (1)  quantitative  and  non- quantitative  heuristic 
techniques  for  conceptualizing  about  human  performance  in  the  context  of 
the  system  environment  and  activity,  (2)  techniques  for  structuring  and 
controlling  observation  and  recording  in  empirical  assessment  situations, 
and  (3)  quantitative  (statistical  and  mathematical)  techniques  for  examining 
the  meaning  of  data  against  expectations  and  needs. 

Many  of  the  techniques  noted  in  this  chapter  have  long  histories  of 
usefulness  in  the  pursuit  of  evaluative  information  concerning  human  per¬ 
formance.  This  is  the  case  for  rating  scales,  questionnaires,  and  some 
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parametric  statistical  techniques 


Noticeable  among  the  relatively  nove  1  and  promising  approaches  are 
various  techniques  of  "modern"  mathematics,  (e,g*»  graph  theory,  game 
theory),  the  use  of  computer-based  "automated  laboratories"*  d  the 
treatment  of  data  by  ncmparametric  statistical  analysis. 

It  was  noted  that  the  major  advantage  of  the  newer,  non-  *  tssical 
mathemat -ai  approaches  is  that  they  can  aid  the  behavioral  *x>  ntist 
discover  and  explore  aspects  of  human  performance  and  conceive  oi  the 
variables,  range  of  variation  and  covariation  in  terms  testabk  by  the 
classical  mathematical  techniques.  In  addition  to  the  use  of  he  uric  tie 
mathematical  procedures,  it  was  noted  that  non- quantitative  he  ristic  proce¬ 
dures  are  available  and  are  in  general  use.  It  would  seem  th  u  a  promising 
direction  for  improving  tne  tools  of  early  system  design  is  the  combination 
of  the  diagrammatic -verbal  and  the  mathematical  heuristics. 

The  use  of  automated  laboratories  is  fairly  widespread  at  this  time. 
These  are  computer-based  stimulus  presentation  and  response  recording, 
scoring  systems.  This  deveiopmen'  probably  represents  a  modern  version 
of  the  "brass-instrument"  experimentation  in  the  behavioral  sciences,  but 
the  promise  of  automated  and  reliable  data  collection  as  well  as  the  handling 
and  on-line  reduction  of  large  amounts  of  information  is  exciting*  The 
possibility  of  new  efficiencies  in  the  matter  of  deciding*  through  computer- 
based  sequential  sampling  techniques,  when  to  stop  collecting  data  is  also 
promising  in  this  context. 

The  review  of  practices  in  data  reduction  indicate  t  e  availability  and 
potential  usefulness  of  nonparametric  statistical  techniques  for  analysing 
measures  of  human  performance.  The  techniques  are  generally  similar  to 
parametric  techniques  in  power  and  use  and  are  practicable  in  system  test 
and  evaluation  programs.  Moreover,  these  techniques  are,  in  many  cases, 
more  appropriate  than  parametric  techniques  for  treating  the  data  of  human 
performance. 
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REVIEW  OF  ASSESSMENT  PRACTICES  IN  SELECTED 
AIR  FORCE  SYSTEMS 


A«  Introduction  ta  the  Review  of  System* 

This  section  presents  &  discussion  and  tabulation  of  practices,  as  re* 
viewed  during  this  study,  for  assessing  human  performance  in  four  major 
types  of  man- machine  systems*  The  four  types  of  systems  are:  space, 
aeronautical,  missile,  and  electronic,  these  designations  deriving  princi¬ 
pally  from  the  name  of  the  division  within  Air  Force  Systems  Command 
which  has  had  cogniaane..*  over  the  development  of  the  system  or  subsystem. 
The  focus  of  this  section  is  on  a  review  of  methods  and  techniques  rather 
than  the  findings  of  the  assessment  programs.  The  purpose  here  is  prin¬ 
cipally  descriptive  rather  than  evaluative,  so  that,  beyond  an  organization 
of  the  information  into  format  convenient  for  tabulation  and  exposition,  no 
attempt  is  made  to  compare  methods  or  techniques  or  to  derive  an  evalua¬ 
tive  schema.  This  information  is  intended  also  to  be  illustrative  rather 
than  definitive.  Clearly,  it  does  not  represent  all  the  programs  that  have 
been  carried  out  to  date,  but  it  does  satisfy  the  general  purpose  of  this  re¬ 
view,  namely,  the  illustration  of  the  ways  in  which  human  performance  is 
known  and  assessed  in  the  context  of  the  typical  military  system. 

This  section  is  subdivided  into  five  aub-s*  .vtions.  Four  of  these  present 
findings  concerning  practices  in  different  types  of  systems.  The  last  sub¬ 
section  summarises  the  information  and  categorizes  the  techniques  illus¬ 
trated  in  the  various  test  reports  and  programs.  A  listing  of  reports  avail¬ 
able  and  used  during  this  review  has  been  organized  by  type  of  system  and 
is  presented  in  the  Reference  section  of  this  report, 

B.  Practices  for  Assessing  Human  Performance  in  Space  Systems 

1,  Human  Performance  in  Space  Systems 

Space  systems  are  designed  to  function  principally  in  the  extra¬ 
terrestrial  environment  although  many  activities  in  the  typical  mission  take 
place  on  the  ground.  Humans  function  generally  in  these  systems  as  vehi¬ 
cle  controllers  and  passengers,  ground  controllers  and  assistants  to  the 
vehicle  pilot(s),  and/or  as  decision-makers  and  participants  in  the  check¬ 
out,  launch,  and  recovery  of  the  space  vehicle.  Some  space  vehicles  are 
unmanned:  their  purpose  is  to  extend  the  capacities  of  the  human  on  the 
ground  to  acquire  information,  to  communicate,  etc,  A  variety  of  human 
performances  must  be  described  and  assessed  in  the  typical  system  de¬ 
velopment  program.  Highlighted  among  the  aspects  of  flignt  performance 
(interactions  within  the  focal  P.E.  A,  unit)  that  require  description  and 
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evaluafitm  a?@  performance  as  a  function  of  weightlessness,  life* support  fa* 
giiities,  tpeoial’.sed  displays  and  controls  and  erew  composition,  With  re¬ 
spect  to  the  checkout  and  launch  of  the  space  vehicle  and  its  ground  support 
equipment,  heavy  emphasis  is  being  given  to  the  study  of  automation- -the 
support  of  the  checkout  crew  by  computer  and  ether  equipment.  Ongoing 
studies  in  this  area  concern  human  decision-making  and  both  the  need  for 
and  acceptance  of  computer  support  by  the  maintenance  and  operational  per¬ 
sonnel,  Human  communication  with  the  computer  is  aiso  a  subject  of  inves¬ 
tigation  with  studies  concentrating  largely  upon  an  information  system  for 
man-computer  symbiosis, 

2,  Summary  and  Description  of  Information  Reviewed 

The  foregoing  discussion  limned  those  aspects  of  human  p 
which  are  currently  being  emphasised  in  apace  programs.  This  section 
summarises  the  systems,  types  of  reports,  etc,,  which  concerned  the  assess¬ 
ment  of  such  performances  «nd  which  were  available  during  this  study. 

Table  IX  summarises  and  describes  this  information, 

Asscssessment  activities  in  four  space  systems,  subsystems,  or 
contexts  were  reviewed?  the  GEMINI  B,  MANNED  ORBITING  LABORA¬ 
TORY  (MOL),  APOLLO,  and  SATELLITE  CONTROL  FACILITY,  Table 
IX  indicates  that  the  system  document  available  for  review  described  ac¬ 
tivities  in  all  phases  except  conceptual.  We  reviewed  documents  describing 
activities  during  the  conceptual  phase.  The  reports  tended  to  concentrate 
either  upon  the  activities  which  take  p*aee  during  the  early  system  develop¬ 
ment  phases  or  proficiency  measurement  during  actual  system  operations. 

The  techniques  described  were  oriented  toward  both  laboratory  and  field 
testing  of  subsystems,  equipments,  and  man-machine  interfaces. 

The  information  which  was  available  for  the  space  systems  was  ac¬ 
quired  during  conferences  with  SPO  and  contractor  personnel  concerned 
with  the  specific  system  or  subsystem  and  from  reports  presenting  test 
plans,  results,  or  general  descriptions  of  the  system  life  cycle.  Table  IX 
indicates  that  conferences  were  held  for  all  systems  covered,  and  that  there 
were  a  number  of  test  planning  and  general  system  documents  available. 

The  principal  source  of  the  methodological  information  reported  for  the 
space  systems  wa  s  test  planning  documentation. 

Information  available  for  the  space  systems /subsystems  varied 
in  terms  of  the  focus  or  objectives  of  the  test  planning  or  evaluation.  In 
many  cases,  it  was  clear  that  human  performance  per  se  was  not  measured, 
e,g,,  in  the  static  review  of  human  engineering  characteristics  of  equipment 
design.  In  these  cases,  human  performance  was  assumed  to  be  related  to 
the  conditions  under  which  it  was  observed  or  to  the  products  and  processes 
that  supported  performance.  Performance  was  being  investigated  as  a 
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funetum  uf  design,  training,  operating  procedures,  personnel  skills,  and 
envirunmental  factors,  The  frequency  with  which  this  wa*  done  it  contained 
in  Table  X,  Many  reports,  e,g, ,  last  plans,  contained  mure  than  one  re¬ 
ported  fftQMt 

in  general,  the  available  information  indicates  that  human  perform¬ 
ance  in  space  systems  was  being  evaluated  principally  is  a  function  cl  equip¬ 
ment  design,  operating  procedures  and  environmental  factors, 

3>  Detailed  Findings*  Concerning  Performance  Assessment 

Practices  in  Space  Systems 

The  preceding  section  has  provided  the  framework  of  the  informa¬ 
tion  available  concerning  the  assessment  of  human  performance  in  space 
systems*  This  section  summarises  the  available  details  concerning  assess¬ 
ment  techniques,  This  information  is  contained  in  Table  X,  The  table  in- 
dieatesi  a)  the  system  nomenclature,  b)  the  focus  upon  human  performance, 
e,g*,  as  a  function  of  design,  represented  by  the  techniques  listed,  n)the 
characteristics  of  the  assessment  situation,  i,e,,  static  or  dynamic  evalua¬ 
tions,  d)  the  types  of  techniques  used  for  data  collection  and  data  reduction/ 
evaluation,  and  e)  the  specific  system  documents  from  which  the  informa¬ 
tion  was  derived  and  the  time  period  within  nhich  they  were  published, 

A  variety  of  techniques  are  used  in  space  systems  or  contexts  for 
evaluating  human  performance.  Information  concerning  systems  in  early 
stages  of  development  indicated  the  usefulness  of  paper  and  pencil  techniques, 
such  as  task  equipment  analyses,  time-line,  and  flow  diagrams  for  collecting 
and  examining  information.  The  early  introduction  into  the  development 
cycle  of  high  fidelity  simulation  in  which  man  performed  was  characteristic 
of  th*  programs  reviewed.  Simulation  wrs  accompanied  by  structured  and 
r«moted  observation  forms  and  apparatus,  e,^>,  video  recording.  re¬ 

duction/evaluation  techniques  could  be  divided  into  quantitative  and  qualita¬ 
tive  techniques.  Quantitative  techniques  ranged  from  simple  analyses  using 
frequency  counts  to  sophisticated  treatments  involving  statistical  compari¬ 
sons  and  tests  for  significance  of  differences  and  relationships.  In  the 
qualitative  category,  the  data  reduction  technique  most  often  used  was  the 
narrative  summary,  i.e.,  verbal  descriptive  discussions,  paragraphs,  or 
tables, 

C,  Practices  for  Assessing  Human  Performance  in  Aeronaut!  sal  Systems 
1 ,  Human  Performance  in  Aeronautical  Systems 

Aeronautical  systems  typically  require  that  humans  control  and 
direct  aircraft  and  associated  equipment  subsystems  through  various  phases 
of  powered  flight  and  take-off/landing  during  a  system  mission.  The 
aeronautical  systems  also  include  ground  support  personnel  avl  equipment. 
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Table  X.  Detailed  Summary  of  Human  Performance  Assessment  Practices  in  Space  System 


Equipment  Design  TR.  Training  PS.  Personnel  Selection.  TO.  Operating  Procedures 
Environmental  Factors  ST.  Static  DY.  Dynamic 


The  assessment  of  pilot  performance  often  involves  attention  to  his  physical 
accommodation,  his  control  behavior,  and  his  survival.  The  performance 
of  ground  personnel  consists  largely  of  maintenance  activities,  and  the  evalua¬ 
tive  focus  is  on  the  safety,  reliability,  and  speed  of  their  performance.  Tech¬ 
niques  and  opportunities  for  developing,  measuring,  and  assessing  human  per¬ 
formance  have  been  developed  over  a  period  of  many  years  and  have  benefitted 
from  a  variety  of  techniques  developed  (e.  g. ,  flight  simulators)  and  programs 
conducted  (e.g. ,  the  evaluation  of  pilot  proficiency)  during  and  since  the 
Aviation  Psychology  Program  of  World  War  II. 

2.  Summary  and  Description  of  Information  Reviewed 

The  foregoing  discussion  indicated  some  of  the  emphases  in  the  eval¬ 
uation  of  the  performance  of  pilots  and  ground  support  personnel  in  aeronautical 
system c.  This  section  summarizes  the  systems,  programs,  and  reports 
which  concerned  the  assessment  of  such  performance  and  which  were  available 
during  this  review.  Table  XI  summarizes  and  describes  this  information. 

Assessmer1  ;es  in  eleven  aeronautical  systems,  subsystems 

or  contexts  were  revl  s  ?he  eleven  systems  or  subsystems  included 
bomber,  cargo,  and  fignic-t  aircraft,  and  one  helicopter  system.  Flight  sim¬ 
ulators  for  the  B-47,  B-58,  and  the  F-lll  aircraft  and  the  Sikorsky  helicopter 
simulator  were  also  reviewed.  The  documents  available  for  review  concerned 
systems  or  subsystems  in  the  acquisition  and  operational  phases.  No  docu¬ 
ments  referred  to  these  or  oth^r  man-machine  interactions  in  earlier  phases 
of  development.  The  techniques  described,  therefore,  were  oriented  toward 
assessment  situations  in  which  the  human  component  could  be  seen  interacting 
with  the  real  or  closely  simulated  system  environment.  The  principal  sources 
for  information  concerning  the  use  of  these  techniques  were  documents  con¬ 
taining  test  plans  or  test  results. 

Evaluative  activities  in  aeronautical  systems  focussed  on  perform¬ 
ance  in  a  variety  of  interactions  involving  personnel -environment- activity. 
Equipment  design,  training,  and  procedural  parameters  were  emphasized. 

3 .  Detailed  Findings  Concerning  Performance  Assessment 

Practices  in  Aeronautical  Systems 

The  preceding  section  has  described  the  information  on  which  this 
section  is  based.  Table  XII  presents  details  concerning  the  assessment  tech¬ 
niques  represented  in  the  available  information.  The  format  of  the  presenta¬ 
tion  is  the  same  as  that  used  for  summarizing  methodological  information 
for  the  space  systems. 

The  tabled  information  suggests  a  variety  of  used  and  useful  data 
collection  techniques,  typical  among  which  are  checklists,  questionnaires, 
rating  scales,  interviews,  and  simulators  which  use  automatic  data  recording. 
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Table  XL  Summary  and  Description  of  Information  Reviewed  Concerning 
the  Assessment  of  Human  Performance  in  Aeronautical  System 


Phase 


Table  XIL.  Detailed  Summary  c*f  Human  Performance  Assessment  Practices 
In  Air  Force  Aeronautical  Systems  (continued) 
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Equipment  Design  TR.  Training  PS.  Personnel  Selection  TO.  Operating  Procedures 
Environmental  Factors  ST.  Static  DY.  Dynamic 
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Table  XII.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
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Equipment  Design  TR,  Training  PS.  Personnel  Selection  TO.  Operating  Procedures 

Environmental  Factors  ST  Static  DY.  Dynamic 


In  addition,  collecting  information  through  the  use  of  standard  AFLG  main¬ 
tenance  and  logistics  forms  yields  useful  Information  concerning  the  type  of 
maintenance  activities  performed  in  the  field,  the  time  spent  in  performing 
them,  and  the  type  of  personnel  participating  In  the  job.  The  information 
appears  useful  in  investigating  the  assignments  of  Air  Force  specialists  and 
in  verifying  the  coverage  of  maintenance  subject  matter  in  the  training  cur¬ 
ricula,  For  the  most  part,  information  from  these  sources  is  reduced  to 
narrative  and/or  statistical  summaries, 

D*  Practices  for  Assessing  Human  Performance  in  Missile  Systems 

1*  Human  Performance  in  Missile  Systems 

The  human  role  in  missile  systems,  particularly  ballistic  systems, 
is  largely  to  maintain  readiness,  monitor,  and  launch.  Modern  missile  sys¬ 
tems  are  operationally  ready  to  react  very  quickly  and  their  countdowns  are 
almost  completely  automated  except  for  procedural,  manual,  and  inter-per¬ 
sonnel  safeguards  against  illegitimate  and  unauthorized  launches.  The  typical 
activities  of  the  system  personnel  involve  monitoring  equipment  status,  se¬ 
curity  of  the  launch  and  control  areas;  and  training  and  communicating  within 
the  context  of  the  User  Command  force  exercises.  Humans  work  in  shelters, 
silo*,  or  capsules,  which  are  relatively  comfortable,  and  which  support  life 
for  normal  and  emergency  conditions.  The  environment  is  nshirt- sleeve,  " 
The  human  controller  does  not  fly  or  otherwise  move  with  the  weapon  vehicle. 
The  emphasis  in  programs  for  assessing  human  performance  is  uoon  human 
vigilance,  fault  detection  and  reporting,  safety  and  reliability  in  maintenance 
activities,  malfunction  diagnosis  and  maintenance  dispatching,  and  man-com¬ 
puter  dialogue.  For  the  most  part,  missile  systems  rely  upon  computers  to 
provide  continuous  monitoring  of  large  amounts  of  information  and  to  supply 
console  or  rack  indications  of  operational  and  maintenance  status. 

The  following  section  reviews  the  assessment  of  human  performance 
in  missile  systems.  The  first  part  provides  an  overview  of  the  type  of  in¬ 
formation  available  and  a  summary  of  methodological  information.  The 
second  part  is  more  oetailed  and  emphasizes  specific  foci  and  techniques 
for  dat«',  collection  and  reduction.  Specific  system  reports  are  also  refer¬ 
enced  in  this  part. 

2.  Summary  and  Description  of  Information  Reviewed 

Information  on  seven  missile  systems  or  related  programs  was  re¬ 
viewed,  All  systems  were  in  the  acquisition  stage  at  the  date  of  the  confer¬ 
ence  and/or  the  report.  Test  plans,  test  reports,  and  general  system  in¬ 
formation  were  gathered  for  all  ballistic  missile  systems  to  date.  Test  and 
evaluation  in  missile  systems  concerned  all  aspects  of  personnel  subsystem 
functioning.  Table  XIII  contains  a  summary  of  the  information  available  for 
the  missile  systems. 
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Table  XUL  Summary  and  Oficriptwn  of  fAfommua  Keyiewed  Concetmn^ 
the  Assessment  of  Human  Performance  in  3di**£U  System# 


3t  Detailed  Findiftffa  Performance  Assessment 

Practices  in  Missile  System* 

Table  XIV  presents*  detailed  information  concerning  the  data  eol= 
lection  and  reduction /evaluative  practices  illustrated  in  mi* tile  *y stems  re* 
viewed  during  this  study.  It  is  probably  safe  to  say  that  human  performance 
assessment  in  missile  systems  during  the  acquisition  stage  has  drawn  heavily 
on  the  repertoire  of  the  behavioral  science  methods  and  thut  all  technique* 
for  securing  and  deriving  meaning  from  information  have  aeon  used  to  date. 
Ballistic  missile  test  programs  at  contractor  facilities  and  at  the  Pacific 
and  Atlantic  Missile  Ranges  include  not  only  the  detailed  requirement  but 
provide  the  opportunity  for  assessing  the  performance  of  system  personnel 
m  near=operation^1  environments.  Systematic  data  collection  situations 
have  been  set  up  for  both  operational  and  maintenance  activities.  Character¬ 
istic  of  field  studies  in  ballistic  missile  test  program-*,  the  system  of  trained, 
non-participant  observer -evaluators  has  been  developed  to  a  point  that  sophis¬ 
ticated  statistical  treatment  of  the  data  secured  by  this  system  is  currently 
being  attempted  on  the  Minuteman  program.  Computer  simulation,  using 
the  IBM/Boeing-developed  General  Purpose  System  Simulation  Model,  is 
being  used  to  examine  the  effects  of  various  personnel  errors  and  decisions 
upon  missile  availability,  maintenance  scheduling  and  force  posture,  Oper¬ 
ability,  maintainability  and  reliability  are  key  human  factor  programs  stimu¬ 
lating  careful  demonstrations  under  controlled  conditions  of  adequate  system 
operation  and  maintenance.  The  test  site  at  Vandenberg  Air  Force  Base  has 
developed  a  resident  Air  Force  capability  to  direct  and  carry  out  the  assess¬ 
ment  cf  human  performance.  Human  performance  is  assessed  actively 
through  Category  III  testing  in  the  operational  phase. 

£•  Practices  for  Assessing  Human  Performance  in  Electronic  Systems 
1 ,  Human  Performance  in  Electronic  Systems 

Most  systems  in  the  invcmtrry  of  the  Air  Force  have  electronic 
components  and  subsystems.  Considering  this,  it  would  be  difficult  to  dis¬ 
tinguish  any  single  group  of  systems  as  "electronic,  "  For  purposes  of  this 
section,  however,  those  systems  under  the  jurisdiction  of  the  Electronic 
Systems  Division  during  their  development  are  listed  in  the  category  of 
electronic  systems.  The  human  role  in  these  systems  ia  typically  to  par¬ 
ticipate  in  missions  variously  involving  communication,  command,  control, 
warning,  direction,  detection,  intelligence,  surveillance,  data  processing 
and  similar  activities.  The  use  of  such  terms  suggests  that  these  man- 
machine  systems  function  to  handle  information  and  communications  in 
several  forms  and  for  several  purposes.  From  a  man-centered  point  of 
viow,  the  equipment  of  the  electronic  systems  serve  to  extend  human  capa¬ 
bilities  and  *  litigate  human  limitations  for  sensing,  processing  and  acting 
upon  information  concerning  tactical  and  other  situations  of  military  in¬ 
terest,  System  personnel  typically  monitor  the  displays  provided  by  system 
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Table  XIV.  Detailed  Summary  cf  Human  Performance  Aa*e**me*st  Practice* 
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Table  XIV.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
in  Air  Force  Missile  Syilems  (continued) 


Equipment  Design  TR.  Training  PS.  Personnel  Selection  TO,  Operating  Procedures 

Environmental  Factors  ST.  Svatic  DY.  Dynamic 
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Equipment  Design  TR.  Training  PS.  Personnel  Selection  TO.  Operating  Procedures 
Environmental  Factors  ST.  Static  DY.  Dynamic 


T&bl6  XIV*  Derailed  Summary  of  Human  Performance  Assessment  Practices 
in  Air  Force  Missile  Systems  (continued) 
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Table  XIV.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
in  Air  Force  Missile  Systems  (continued) 


i 


i 
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sensors,  (e,g. ,  radar),  and  are  required  to  query  or  communicate  with  com¬ 
puter  aids  to  human  decision  making.  One  of  the  requirements  of  and  on 
system  personnel  is  therefore  to  relate  closely  to  She  computer  subsystems 
of  the  electronic  system.  The  assessment  of  human  performance  in  these 
systems  typically  focuses  upon  problems  of  human  vigilance,  organization 
of  groups  for  effective  decision-making,  human  information  handling,  de¬ 
cision- making,  and  conditions  for  the  display  of  information  and  effective 
workplaces, 

2.  Summary  and  Description  of  Information  Reviewed 

Table  XV  summarises  the  twelve  systems  surveyed  and  the  type 
and  focus  of  the  information  available  for  this  review  of  assessment  tech¬ 
niques  in  electronic  systems.  No  reports  or  other  materials  were  available 
for  evaluative  activities  that  occurred  in  the  Conceptual  or  Definition  phases 
of  system  life.  Reports  of  assessment  plans  and  activities  occurring  in  the 
Acquisition  stage  were,  however,  available  for  all  systems.  Category  II/III 
test  reports  were  available  for  41 6L  (SAGE)  and  the  AN/GLR-1  portion  of 
the  466L  system.  Evaluative  activities  in  all  systems  related  system  and 
human  performance  to  equipment  design,  operating  (and  maintenance)  pro¬ 
cedures,  and  various  environmental  factors  (e.g.,  illumination,  sound 
levels).  Training  and  personnel  selection  interests  were  mentioned  less 
frequently  in  the  reports  reviewed. 

3.  Detailed  Findings  Concerning  Performance  Assessment 

Practices  in  Electronic  Systems 

The  assessment  of  human  performance  in  electronic  systems  ap¬ 
pears  to  be  well  organized  and  to  utilize  a  wide  variety  of  techniques,  as 
illustrated  in  Table  XVI.  The  typical  set  of  data  collection  procedures  in¬ 
volves  direct  observation  of  human  performance  by  observer /evaluators 
(0/Efs),  the  use  of  data  collection  forms  such  as  checklists,  questionnaires, 
and  rating  scales,  the  use  of  interviews,  and  some  form  of  instrumented 
evaluation  of  environmental  factors.  Other  techniques  which  appear  to  be 
used  less  often  but  nevertheless  are  quite  promising  in  electronic  systems 
include  automatic  data- recording  using  both  computer  programs  to  simulate 
missions  and  record  system  actions,  and  various  forms  of  time  and  event 
recording  apparatus.  In  addition,  other  unusual,  (i.e.,  infrequently  occur  - 
ring),  techniques  include  time  and  motion  studies,  photography,  and  the 
use  of  the  AFTO  210,  211  forms  for  collection  maintenance  data. 

The  promise  of  using  the  built-in  data  collection,  reduction,  and/ 
or  reporting  capacities  of  modern  electronic  systems  should  be  noted  in 
connection  with  this  summary.  Systems  such  as  the  466L  (AN/GLR-1 
portion)  and,  more  specifically,  computer-based  SAGE  and  BUIC  have  the 
capability  of  sensing  and  recording  many  overt  control  actions  of  the  opera¬ 
tors  and  of  tabulating /summarizing  the  information  in  a  variety  of  useful 
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Table  XV.  Summary  and  Description  of  Information  Reviewed  Concerning 
♦be  Assessment  of  Human  Performance  in  Electronic  Systems 


Table  XVI.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
In  Air  Force  .Electronic  Systems 
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Table  XVI.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
In  Air  force  Electronic  Systems  (continued) 


Table  XVI.  Detailed  Summary  of  Human  Performance  Assessment  Practices 
In  Air  Force  Electronic  Systems  (continued) 
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formats,  Th?s  capacity,  carefully  developed  and  provided  for  these  systems, 
does  not,  however,  appear  to  be  used  fully  when  the  systems  become  opera* 
tional*  Some  reasons  given  issV  this  refer  to  Air  Force  doctrine  and/or 
management  policies,  but  it  was  also  noted  that  there  is  a  need  to  develop  a 
nWue*suitn  Air  Force  capability  to  use  imaginatively  the  computer-based 
opportunities  and  techniques  for  assessing  human  performance  in  electronic 
systems. 


Summary  of  Assessment  Practices 


The  preceding  review  of  evaluative  practices  in  thirty-four  Air  Foret  and 
related  programs  illustrates  a  variety  of  techniques  for  acquiring  information 
concerning  human  performance  and  of  deriving  meaning  from  the  information* 
The  tabulations  of  these  various  techniques  having  buen  made  separately  for 
each  system,  the  purpose  here  is  to  provide  a  brief  summary  across  systems. 
The  table  and  figures  which  follow’  contribute  to  this  overview. 


Thirty- four  practices  were  identified  in  the  review  of  the  thirty-four 
assessment  programs.  The  test  plans,  test  reports,  and  discussions  concern¬ 
ing  these  programs  indicated  two  general  categories  of  techniques;  those  used 
in  the  collection  and  recording  of  data,  and  practices  used  for  data  reduction 
and  analysis.  The  data  collection  practices  are  variations  of  indirect  cr  direct 
observation  in  which  reports,  interviews,  and  empirical  situations  such  as 
field  trials,  simulator  studies,  and  experiments  are  used  as  information  sources 
or  test  situations.  Human  observation  is  often  instrumented  or  structured 
through  the  use  of  programmed  materials:  checklists,  rating  scales,  apparatus 
set  to  measure  selected  conditions  at  pre-selected  data  points,  etc.  Often, 
too,  the  observer  is  remoted  through  the  use,  of  on-line  video  and  sound  mon¬ 
itors,  and  video  and  sound  recordings  for  off-line,  post- test  analysis.  The 
data  reduction  techniques  illustrated  in  the  system  situations  of  this  study 
most  often  involved  narrative  summaries  in  a  variety  of  formats  and  the  use 
of  descriptive  statistics  (means,  medians  and  associated  estimates  of  vari¬ 
ability),  In  some  cases,  test  data  were  manipulated  statistically  in  order  to 
test  hypotheses,  interrelating  sets  of  data  and/or  checking  the  statistical  sig¬ 
nificance  of  results.  In  a  relatively  few  cases,  expert  judgment  was  given  as 
the  principal  means  of  deriving  meaning  from  the  assessment  information 
concerning  human  performance.  Table  XVII  correlates  information  concern¬ 
ing  assessment  techniques  with  the  specific  Air  Force  systems  on  which  test 
information  was  available  during  this  study. 


The  summaries  graphically  presented  in  Figures  7  and  8  are  derived 
from  Table  XVII*  Figure  7  makes  visible,  in  summary  form,  the  frequency 
of  reported  use  for  each  assessment  technique.  Interview  and  observation 
techniques  were  named  in  practically  all  test  plans  and  reports  and  were 
supported  by  various  other  techniques  that  combined  interview  and  observa¬ 
tion  with  paper  and  pencil  check  or  work  sheets  and  recording/measuring 
apparatus.  Narrative  and  descriptive  statistic  summaries  were  most  fre¬ 
quently  noted  in  the  test  plans  and  reports  reviewed. 
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Tabic  XV n,  SommaT,  of  >«s*a**a*at  Practice*  Reported  is  the  Startem*  a»d  S»b***tes»*  ate^v^ 
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Figure  7.  Frequency  of  Reported  Uses  of  Human  Performance 

Assessment  Practices  in  the  Systems  Reviewed  During 
This  Study 
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From  Figure  8  it  is  possible  to  make  some  comparisons  among  the  test 
situations  surveyed  during  this  study.  The  information  made  available  by 
Air  Force  SPO*s  and  contractors  indicated  that  the  assessment  of  human  per¬ 
formance  in  missile  systems  utilizes  the  largest  number  of  the  different  tech¬ 
niques  described  in  Table  XVII,  The  space,  aeronautical,  and  electronic 
systems  noted  about  the  same  number  of  useful  assessment  techniques. 
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CONCLUDING  REMARKS 


A.  Synopsis  of  Report 

This  report  was  written  as  a  review  of  practices  in  the  assessment 
of  human  performance  in  Air  Force  systems.  The  review  was  carried 
out  during  the  preparation  of  a  methodological  handbook  on  the  assessment 
of  human  performance.  In  order  to  obtain  and  bring  together  the  most 
inclusive  overview  of  the  topic  we  found  it  necessary  to  (1)  define  opera¬ 
tionally  both  human  performance  and  the  assessment  of  human  performance 
in  the  systems  context,  (2)  review  the  current  Air  Force  policies  and 
practices  for  developing  and  testing  military  systems  and  for  developing 
and  evaluating  personnel  subsystems,  (3)  review  the  methodologies  of  the 
behavioral  sciences  relating  to  human  performance  in  the  world  of  work, 
and  (4)  survey  practices  in  Air  Force  system  programs  for  studying  human 
performance  and/or  the  variables  of  which  human  performance  is  a  function. 
Figure  9  summarizes  these  steps  and  the  content  of  this  report. 

B.  General  Comments  Concerning  Human  Performance 

Our  perspective  herein  has  been  that  the  reality  that  basically  concerns 
the  system  scientist,  behavioral  or  otherwise,  in  the  development  of  an 
operable  and  maintainable  military  system  is  selected  system  PERSONNEL 
in  the  system  ENVIRONMENT  (which  includes  system  equipment,  ambient 
conditions  and  system  personnel  other  than  those  in  the  focus  of  attention) 
carrying  out  some  system  ACTIVITY  or  procedure.  These  personnel- 
environment-activity  dimensions  and  interactions  among  them  constitute 
the  subject  matter  of  inquiries  into  human  performance  in  Air  Force  systems. 
Accordingly,  the  subject  matter  breaks  down  into  five  anthropocentric  areas: 
individual  human  capacities  and  limitations,  the  interaction  of  system  personnel 
with  each  other,  the  interaction  of  system  personnel  and  system  equipment, 
the  interaction  of  system  personnel  and  the  ambient  system  conditions,  and 
the  interaction  of  system  personnel  with  the  activities,  procedures  and  work 
of  the  system.  While  this  perspective  may  oversimplify  the  scope  of  human 
performance  studies,  it  nevertheless  see  ms  to  be  useful  for  structuring  a 
review  of  the  specialists  involved  in  the  study  of  human  performance  in 
system  contexts  and  the  disciplines  and  technologies  within  the  behavioral 
sciences  applicable  to  the  assessment  of  human  performance. 

G.  General  Comments  Concerning  Assessment 

The  assessment  of  human  performance  has  been  considered  here  as 
the  process  of  analyzing  and  describing,  collecting  information  concerning. 
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Figure  9.  Content  Summary  of  This  Report 
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and  evaluating  human  performance  (conceived  as  interactions  among  system 
personnel,  system  environment,  and  system  procedural  dimensions).  The 
process  includes  the  human  acquisition  of  information  through  logical  assump¬ 
tion,  indirect  observation,  and  direct  observation.  The  evaluation  of  the  infor¬ 
mation  requires  the  clear  specification  of  the  need  and  use  for  the  informa¬ 
tion  and  the  comparison  of  what  is  obtained  against  what  is  required,  or 
previously  understood,  or  previously  not  known.  The  use  of  the  evaluated 
information  very  often  requires  consensus  among  the  system  designers  and 
users  concerning  its  value  and  meaning  to  system,  mission  performance. 

The  practice  of  this  assessment  methodology  is  well  guided  by  rules 
and  advice  from  both  military  and  behavioral  science  sources.  The  tech¬ 
nical  decisions  as  to  what  performance  information  is  needed  and  whether 
a  test  situation  is  required  must,  nevertheless,  be  made  by  the  skilled 
specialist, 

D,  General  Comments  Concerning  Assessment  Technology 

The  assessment  methodology  just  described  is  supported  by  a  repertoire 
of  useful  techniques  for  getting  to  know  about  and  deriving  meaning  from 
the  knowledge  concerning  human  performance  in  Air  Force  systems.  The 
behavioral  science  technology  contains  numerous  techniques  that  can  be  used 
in  the  conceptual  (i.  e,  f  the  human  performance  is  conceptual)  and  the  empirical 
(i.  e, ,  the  human  performance  is  observable)  assessment  situations.  The 
reliability  and  validity  of  the  resulting  information  vary  and  depend  largely 
upon  the  care  with  which  the  test  situation  and  instruments  a^e  developed. 

Most  of  the  techniques  available  and  used  in  the  behavioral  sciences 
have  been  applied  to  assessing  human  performance  in  Air  Force  systems. 

These  include  a  variety  of  techniques  for  data  collection  and  reduction/ 
analysis.  The  developments  which  are  most  promising  for  application  to 
the  assessment  of  human  performance  in  Air  Force  systems  involve  the 
quantification  of  verbal  and  diagrammatic  heuristic  techniques,  the  use  of 
automated  laboratories,  and  the  more  widespread  use  of  nonparametric 
statistical  treatments. 


APPENDIX  I 


SCHEMA  FOR  REVIEW  OF  BEHAVIORAL  SCIENCE  METHODS 
FOR  THE  MEASUREMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE 


Major  Field  No*  1  -  Human  Performance  Studies 


Basic  Research  Subfields 


Applied  Research  Subfields 


individual  Performance 
Aging 

Anthropometry 
Decision  Processes 
Environmental  Effects  on 
Performance 

Atmospheric  Properties 
Environmental  Uniformity 
Gravitation  and  Inertia 
Magnetic  Fields 
Radiation 
Thermal  Effects 
Vibration  and  Blast 
Visible  Spectrum 
Motivation  and  Stress 
Motor  Performance 
Perceptual  Performance 
Personality  and  Character 
Correlates 

Research  Methodology  and  Theory 
Task  and  Vigilance  Fatigue 
Other 

Group  Performance 

Group  Communication  Prccesses 

Group  Decision  Processes 

Group  Dynamics 

Groap  Influence  on  Perception 

Group  Standards  and  Performance 

Group  Structure 

Other 


Human  Engineering 

Clothing  and  Personal  Equipment 
Command  and  Control 
Communication 
Controls 

Design  in  Relation  to  the  Availability 

of  Human  Resources 

Display  of  Information 

Equipment  Acceptability 

Human  Engineering  Bibliographies 

and  Handbooks 

Information  Processing 

Layout  of  Work  Places 

Maintenance 

Man- Machine  Dynamics 

QOPRI  Methodology 

Reconnaissance  Technology 

Theory  Methodology  and  Apparatus 

Other 

Team,  System  Performance 
Crew  Composition 
Large  Organization  Structure  and 
Individual  Effectiveness 
Organization  Effectiveness 
Other 


SCHEMA  FOR  REVIEW  OF  BEHAVIORAL  SCIENCE  METHODS 
FOR  THE  MEASUREMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE  (continued) 


Major  Field  No.  2  -  Personnel  Research  Studies 


Basic  Research  Subfields  Applied  Research  Subfields 


Individual  Differences 
Criterion  Theory 
Predictor  Theory 
Psychometrics 
Other 

Learning  and  Retention 
Correlates  of  Learning  Proficiency 
Group  Factors  in  Learning 
Human  Learning  Activities 
Learning  Theories  and  Compara¬ 
tive  Learning 
Motivation  in  Learning 
Retention 

Transfer  of  Training 
Other 


Selection  and  Assignment 
Assignment,  Allocation  and  Dis¬ 
tribution  Procedures 
Career  Guidance 
Classification  Procedures 
Criteria  Development 
Performance  Evaluation  Techniques 
Predictor  Techniques 
Psychiatric  Selection  Research 
Relation  to  Training  Requirements 
U.S*  Military  Sociology 
Other 

Training 

Evaluating  Training  Effectiveness 
Methods  of  Studying  Training 
Requirements 

Task  and  Skill  Analyses  Methods 
Training  Methods 

Programmed  Instruction 
Training  Aid  Design 
Training  Media  Effectiveness 
Curriculum  Planning 
On-the-Job  and  In-Class 
Training  Procedures 
Other 
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SCHEMA  FOR  REVIEW  OF  BEHAVIORAL  SCIENCE  METHODS 
FOR  THE  MEASUREMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE  (continued) 


Major  Field  No»  2  -  Personnel  Research  Studies  (continued) 


Basic  Research  Subfields 


Applied  Research  Subfields 


Per  s 


Adapting  Available  Tests  to  Opera¬ 
tional  Situations 

Design  of  Specific  Test  Batteries 
Determining  Program  Training 
Requirements 

Individual  Proficiency  Training 
Leadership  Training 
Performance  Evaluation  Tests 
Recruitment 

Selection  and  Training  for  Remote 
Area  Operations 

Specific  Training  Aid  and  Simulator 
Design 

Team  and  System  Training 
Test  Maintenance 
Training  Foreign  Nationals 
Other 


Major  Field  No,  3  -  Human  Support  and  Maintenance  Studies 


Bionics 
Bio  simulation 

Techniques  of  PsychophysioLogical 
Monitoring 

Electrical  Stimulation  Equipment 
Other 


Maladjustment 
Accidents 
Adjustment 
Deviant  Behavior 

Character  Disorders 
Cultural  Factors 
Mental  Illness 
Morale 
Other 


SCHEMA  FOR  REVIEW  OF  BEHAVIORAL  SCIENCE  METHODS 
FOR  THE  MEASUREMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE  (continued) 


Major  Field  No«  3  -  Human  Support  and  Maintenance  (continued) 


Basic  Research  Sublie  Ids  Applied  Research  Subfields 


Ecology 

Terrestrial 

Aquatic 

Atmospheric 

Other 

Psychophysiology 
Activation  Systems 
Biochemical  Processes  and  Drugs 
Environment  Effects  on  Comfort, 
Health  and  Safety 

Atmospheric  Properties 
Environmental  Uniformity 
Gravitation  and  Inertia 
Magnetic  Fields 
Radiation 
Thermal  Effects 
Vibration  and  Blast 
Visible  Spectrum 
Ecological  Studies 
Homeostatic  Systems 
Neurological 

Psychophysiological  Correlates  of 

Emotion 

Sensory 

Sleep 

Other _ 


Protective  Devices 

Clothing  and  Personal  Equipment 

Motion  Sickness 

Other 

Support  Devices 
Design  of  Space  Platform 
Escape  and  Evasion 
Prosthetic  Devices 
Shelter  Habitability 
Other 


SCHEMA  FOR  REVIEW  OF  BEHAVIORAL  SCIENCE  METHODS 
FOR  THE  MEASUREMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE  (continued) 


Major  Field  No,  4  -  Economic  Analyst*  end  Management  Studies 


Basic  Research  Subfields 


Theory 

Decision  theory 
Economic  theory 
Game  theory 
Statistical  methods 
Graph  theory 


Applied  Research  Subfields 


Techniques  and  Methods 
Network  analysis  (PERT,  etc* ) 
Econometric s** -methods,  models, 
theory 
Gaming 

Process  analysis 
Value  engineering 
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LISTING  OF  AIR  FORCE  REQUIREMENT  DOCUMENTS 
STRUCTURING  THE  DEVELOPMENT  AND  ASSESSMENT  OF 
HUMAN  PERFORMANCE 

Air  Force  Regulations 

Document 

Number 

Subject 

AFR  0-2 

Numerical  Index  of  Standard  Air  Force  Publications 

AFR  0-6 

Subject  Index  of  Air  Force  Publications 

AFR  23-2 

Air  Force  Logistic  Command 

AFR  23-6 

Air  Training  Command 

AFR  26-1 

Manpower  and  Organization  Activity 

AFR  26-3 

Manpower  Authorization,  Policy  and  Procedures 

AFR  30-5 

Guidance  to  Insure  Adequate  Personnel  Facilities 

AFR  30-8 

Development  of  a  Personnel  Subsystem  for  Aerospace 
Systems 

AFR  35-14 

Attending  Meetings  of  Technical,  Scientific,  Pro¬ 
fessional  or  Similar  Organizations 

AFR  40-423 

Non- Government  Training 

AFR  50-3 

USAF  Training  for  AF  Contractor  Employees 

AFR  50-9 

Special  Training 

AFR  50-19 

Management  of  Training  Equipment 

AFR  53-12 

USAF  Instrument  Pilot  Instructor  School 

AFR  57-4 

Modification/Modernization  of  Aircraft,  Guided 
Missiles  and  Related  Equipment 

AFR  58-4 

Responsibilities  for  Missile  /Space  Accident  Pre¬ 
vention  Programs 

201 


Document 

Number 

AFR  65-3 

AFR  66-1 

AFR  66-7 

AFR  66-8 

AFR  66-18 
AFR  66-29 

AFR  66-30 
AFR  67-19 

AFR  80-5 

AFR  80-6 
AFR  80-11 
AFR  80-14 

AFR  80-24 
AFR  80-27 
AFR  80-28 
AFR  80-31 

AFR  80-32 
AFR  80-36 


Subject 

Configuration  Management 

Policy,  Objectives  and  Responsibility 

Technical  Order  System 

Maintenance  Evaluation  Program- -Vehicles  and 
Aerospace  Ground  Equipment 

Contract  Technical  Services  (CTS) 

Maintainability  Program  for  Weapon  Support  and 
Command  and  Control  Systems 

Product  Improvement  Program 

Supply  Support  of  Research,  Development,  Test 
and  Evaluation  Activities 

Reliability  Program  for  Systems,  Subsystems, 
and  Equipment 

Classification  of  Air  Force  Equipment 
Importance  Categories 

Testing  Evaluation  of  Systems,  Subsystems, 
and  Equipments 

Test  Results  of  Commercial  Equipment 
Research  and  Development  Plans  and  Reports 
Engineering  Inspections 

Elimination  Program  for  Air  Force  Materiel  Within 
the  Sensible  Atmosphere 

Quick  Reaction  Capability 

Civil  Airworthiness  Standards  for  USAF 
Transport  Aircraft 
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Document 

Number 

AFR  122-4 

AFR  160-3 

AFR  161-2 

AFR  310-1 

AFR  375-1 

AFR  375-2 

AFR  375-3 

AFR  375-4 

AFR  400-25 

AFR  400-26 

AFM  11-1 

AFM  26-1 
AFM  32-3 
AFM  35-1 
AFM  36-1 
AFM  64-4 
AFM  66-1 

AFM  66-18 

AFM  110-9 
AFM  172-1 


Subject 

Nuclear  Safety  -  The  Two  Man  Concept 

Hazardous  Noise  Exposure 

Aerospace  Systems  Management  Medical 

Management  of  Contractor  Data 

Management  of  Systems  Program 

System  Program  Office 

System  Program  Director 

System  Program  Documentation 

Logistics  Support  for  Other  than  Categories 
I,  II,  and  III  Tests 

Logistics  Support  for  Weapon,  Support,  and 
Control  Systems  Test  Programs 

Air  Force  Manuals 

AF  Glossary  of  Standardized  Terms  and 
Definitions 

Policies,  Procedures  and  Criteria 

Ground  Safety  -  Accident  Prevention  Handbook 

Airman  Classification  Manual  (Vol  I  and  II) 

Officer  Classification  Manual 

Handbook  for  Personal  Equipment,  Personnel 

Maintenance  Management  -  Depot,  Field  and 
Organizational  Maintenance 

Contractor  Technical  Services  for  Category  II 
and  III 

System  Program  Documentation 
Budget  Administration 
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Document 

Number 


Subject 

Air  Force  Systems  Command  Regulations 


AFSCR  11-2 
AFSCR  80-20 

AFSCR  80-4 
AFSCR  80-16 

AFSCR  66-7-8 

AFSCR  80-23 


AFSCPMI  1-4 
AFSCPMI  1-5 
AFSCPMI  2-5 
AFSCPMI  2-7 
AFSCPMI  4-2 
AFSCPMI  4-3 

AFSCPMI  4-7 
AFSCPMI  6-8 

AFSCPMI  6-10 


Presto  Reporting  System 

Research  and  Development  -  Air  Force  System 
Command  Technical  Report  Program 

Status  Classification  of  Air  Force  Equipment 

Personnel  Subsystem  Program  fc^  Aerospace, 
Support,  and  Command  and  Control  Systems 

Depot,  Field  Organizational  Maintenance 
Technical  Order  Data 

Gold  Weather  Test  Responsibility 

Air  Force  Systems  Command  Program 
Management  Instructions 

System  Package  Program  Format 
Test  Report 

Advanced  Development  Program 
Job  Plan  Format  (superceded  by  AFSCPMI  2-5) 
In  Service  Engineering  Procedures 
Engineering  Support  Procedures 

Engineering  Service  Plan  Format 
Status  Classification 

Test  and  Evaluation 


104 


Document 

Number 

AFSCM  5-1 

AF55CM  80-1 

AFSCM  80-3 

AFSCM  80-5 

AFSCM  80-6 

AFSCM  80-7 

I 

AFSCM  80-8 

AFSCM  80-9 

AFSCM  122-1 
AFSCM  375-1 

AFSCM  375-4 
AFSCM  375-5 
AFSCM  310-1 

I 


Subject 

Air  Force  System  Command  Manuals 

Research  and  Development  -  Air  Force  System 
Command  Technical  Report  Program 
(superseded  by  AFSCR  80-20) 

Handbook  of  Instructions  for  Aircraft  Designers 
Vols.  I  and  III  (HIAD) 

Handbook  of  Instructions  for  Aerospace 
Personnel  Subsystem  Designers  (HIAPSED) 

Handbook  of  Instructions  for  Ground 
Equipment  Designers  (HIGED) 

Handbook  of  Instructions  for  Aircraft  Ground 
Support  Equipment  Designers  (HI AGED) 

Handbook  of  Instructions  for  Aerospace 
Vehicle  Equipment  Design  (HIAVED) 

Handbook  of  Instructions  for  Missile  Designers, 
Vols.  I  and  H  (HIMD) 

Handbook  of  Instructions  for  Aerospace 
Systems  Design  (HIASD) 

The  Nuclear  Weapons  Safety  Program 

Configuration  Management  During  the 
Definition  and  Acquisition  Phases 

System  Program  Management  Manual 

System  Engineering  Management  Procedures 

Management  of  Contractor  Data  and  Reports 


T 
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Document 

Number 


AFBM  57-8A 

AFBM  58-1 
AFBM  58-9 

AFBM  58-10 

AFBM  59-17 

AFBM  58-18C 

AFBM  60-1 

AFBM  60- 65 A 

AFBM  59-32 

AFBSD  61-99 

AFBSD  62-41 

AFBSD  62-53 
AFBSD  62-79 
ESDP  375-1 


Subject 

AFSC  Division  Exhibits 

Human  Engineering  Design  Standards  for 
Missile  System  Equipment 

Contractor  Report 

Inspection  Requirements  Manuals,  Inspection 
Cards  and  Sequence  Charts  for  Ballistic 
Missile  Weapon  Systems 

Reliability  Program  for  Ballistic  Missile  and 
Space  Systems 

Training  Equipment  Procurement  for  Air  Force 
Ballistic  Missiles  and  Military  Space  Systems 

Quantitative  and  Qualitative  Personnel  Require¬ 
ments  Information 

Personnel  Subsystem  Testing  for  Ballistic 
Missile  and  Space  Systems 

Aerospace  System  Personnel- -Equipment  Data 
for  Personnel  Subsystem  Development 

Design  for  Maintainability  Program  for  Weapon 
and  Space  Systems 

Human  Engineering,  Development  of  System, 
General  Specifications  for 

System  Safety  Engineering:  Genera’  Specifications 
for  the  Development  of  Air  Force  Ballistic 
Missile  Systems 

WS-133B  Maintainability  Design  Criteria 

Life  Support  Subsystem  Criteria  (WS-133B) 

A  Typical  Test  Section  of  a  System  Package 
Program  for  an  Electronic  System 
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n 


Document 

Number  -Subject 

WCLDPT  60-21  A  Technical  Guide  for  Designers  of  Personnel 
Subsystems  for  Weapon/Support  Systems 

WDT  56-5  Technical  Manual  Program 

Military  Standards 

MIL- STD- 105  Sampling  Procedures  and  Tables  for 

Inspection  by  Attributes 

MIL-STD-218  Technical  Manuals 


MIL-  STD- 441 
MIL- STD- 8  03  A-l 

MIL- STD- 831 


MIL-  T- 4857 


MIL-M-5474 


MIL-H-6814 


MIL-T-9107 
MIL- D- 93 10 

MIL-  W- 9411 

MIL- D- 9412 


Reliability  of  Military  Electronic  Equipment 

Human  Engineering  Design  Criteria  for 
Aerospace  Ground  Equipment 

Preparation  of  Test  Reports 

Military  Specifications 

Training  Equipment,  Weapon  System,  Specification 
and  Specification  Compliance  Test  Outlines,  In¬ 
structions  and  Requirements  for  Preparation  of 
(^Superseded  by  MIL-T-27382) 

Technical  Manuals,  General  Requirements  for 
Preparation  of 

Handbooks,  Overhaul,  Electronic,  Electrical 
Electro-Hydraulic,  Electro- Mechanical  Equip¬ 
ments,  Systems,  and  Test  Equipments, 

Preparation  of 

Test  Reports,  Preparation  of 

Data  for  Aeronautical  Weapon  Systems  and 
Support  Systems 

Weapon  Systems,  Aeronautical,  General 
Specification  for 

Data  for  Aerospace  Ground  Equipment  (AGE) 
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Document 

Number 

MIL- M- 9864 
MIL-  C« 9883  A 

MIL-H-  25946 
MIL- P-25996 

MIL-H-26207 

MIL- D- 26239 

MIL- M- 2 65 12 

MIL-S-26634 

MIL- R- 26667 

MIL-  R-  26674 

MIL-T-27382 

MIL-T-27474 

MIL- R- 27542 


Subject 

Technical  Manuals:  Operation  and  Organizational 
Maintenance  (Missile  Weapon  System) 

Check  Lists  for  Missile  and  Space  Systems 
Operational  and  Oi  ganizational  Maintenance 

Human  Factors  for  Manned  Aircraft  Weapon 
Systems  (Superseded  by  MIL-H-27894) 

Procedures  for  the  Development  of  a  Cockpit 
Subsystem  and  the  Accomplishment  of 
Subsystems  Integration 

Human  Factors  Data  for  Guided  Missile  Weapon 
Systems  (Superseded  by  MIL-H-27894) 

Data,  Qualitative  and  Quantitative  Personnel 
Requirements  Information  (QQPRI) 

Maintainability  Requirements  for  Aerospace 
Systems  and  Equipment 

Specifications,  Weapon  System,  and  Support 
System  Mock-ups,  Preparation  of 

Reliability  and  Longevity  Requirements,  Electronic 
Equipment,  General  Specifications  for 

Reliability  Requirements  for  Weapons  Systems 
(Superseded  by  MIL-R-27542) 

Training  Equipment,  Subsystem,  Technical  Data, 
Preparation  of  (Supersedes  MIL-T-4857) 

Training  Equipment,  Ground,  General 
Requirements  for 

Reliability  Program  Require: nents  for  Aerospace 
Systems,  Subsystems  and  Equipments 
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Doeument 

Numbtf 

MIL-T-276I5 

MIL-H-27894 

MIL-S-38130 

MIL- M- 38701 

MIL- S- 58077 

MIL- D- 70327 


Subject 

Test  Outline,  Engineering,  lor  the  Inspection 
of  Training  Equipment,  Requirements  for  the 
Preparation  of 

Human  Engineering  Requirements  for  Aerospace 
Systems  and  Equipment 

Safety  Engineering  of  Systems  and  Associated 
Subsystems,  and  Equipment,  Genaral  Requirements  for 

Manuals,  Technical,  Inspection  Requirements, 

Work  Cards,  for  Missile  and  Space  Weapon 
Systems 

Safety  Engineering  of  Aircraft  Systems, 

Associated  Subsystems  and  Equipments; 

General  Requirements  for 

Drawings,  Engineering  and  Associated  Lists 
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APPENDIX  III 


DEVELOPMENT  OF  PERSONNEL  SUBSYSTEMS  IN  AIR 
FORCE  SYSTEMS;  AIR  FORCE  POLICY,  REQUIREMENTS 
AND  DEFINITIONS  (SOURCE;  AFR  30.8) 


Scope  and  Policy 


Scope; 

This  regulation  applies  only  to  systems  and  programs  managed  in  accorc 
ance  with  AFRa  in  the  375  series*  However,  other  USAF  development  pro¬ 
grams  must  be  planned  and  implemented  with  full  consideration  for,  and 
integration  of,  the  applicable  elements  of  personnel  subsystems  described 
herein* 

USAF  Policy: 

a.  Manpower,  personnel,  and  training  actions  must  be  defined,  scheduled, 
and  performed  in  a  coordinated  manner,  compatible  with  all  other 
aspects  of  system  development  in  order  to  provide  qualified  personnel 
at  a  predetermined  time  and  place. 

b.  Required  leadtimes  for  PSEs  will  be  determined  and  considered  in  the 
overall  system  planning*  The  SPD  will  advise  HQ  USAF  of  any  situa¬ 
tion  that  may  prevent  meeting  the  approved  Sys  em  Master  Schedule. 

c*  Contractor- furnished  reports  in  the  personnel  and  training  PSEs  will 
only  be  developed  to  meet  requirements  justified  by  major  commands 
on  a  valid  Mneed  to  have"  basis.  Simplicity  and  austerity  wiJl  be  the 
rule  in  development  of  such  data.  Duplication  of  data  effort  and  un¬ 
necessary  information  and  detail  will  be  avoided  (see  AFR  310-1). 

d.  The  PSEs  described  in  this  regulation  are  interdependent  and  will  be 
developed  concurrently  whenever  possible.  Preliminarv  elements 
will  be  refined  as  permitted  by  development  of  the  system* 

e.  The  PS  will  be  provided  priorities  compared  with  other  elements  of 
the  system  or  program. 

f.  Deviations  which  facilitate  development  of  the  necessary  PSEs,  or 
save  time  or  money,  will  be  referred  to  HQ  USAF  for  approval  after 
coordination  with  all  interested  commands  /agencies. 
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•»  AFSGM  80-3,  Handbook  of  Instructions  for  Aerospace  Personnel  Sub¬ 
system  Designers  will  be  the  primary  source  of  information  and 
guidance  for  PS  implementation,  to  the  extent  it  does  not  conflict  with 
this  regulation. 


— Personnel  Subsystem  Elements 

The  following  PSEs  are  usually  essential  in  developing  a  system  or 
program.  The  details  of  a  PSE  and  its  associated  management  milestones 
will  differ  for  each  program  and  must  be  defined «  scheduled,  prepared, 
integrated,  coordinated  and  accomplished  as  a  team  effort  according  to 
AFR  375-2,  No  rigid  sequence  is  intended  in  the  order  of  listing  of  the 
PSEs  as  each  is  interdependent,  and  concurrent  development  is  the  rule 
rather  than  the  exception, 

a,  P»rsonnel/Equipment  Data  (PED): 

PED  is  centrally  controlled  analytical  da':a  in  the  form  of  task  and 
equipment  information.  It  defines  the  relationship  between  system 
personnel  and  system  hardware,  other  PSEs,  and  the  technical  data 
requirements  of  AFR  310-1,  PED  is  basic  to,  and  provides  neces¬ 
sary  data  for,  preparing  and/or  developing  other  PSEs, 

b.  Human  Engineering: 

Human  engineering  is  the  application  of  knowledge  of  man's  capabilities 
and  limitations  to  the  planning,  design,  development,  and  testing  of 
aerospace  systems,  equipment,  and  facilities  to  achieve  optimum  per¬ 
sonnel  safety,  comfort,  and  effectiveness  compatible  with  system 
requirements.  It  includes  participation  in  the  identification  of  require¬ 
ments  for,  and  in  the  design,  development,  and  testing  of,  operator  and 
maintenance  crew  stations,  personnel  environments,  layouts,  controls, 
displays,  job  procedures  and  performance  aids.  Human  engineering 
also  provides  basic  information  and  data  required  in  the  development  of 
ether  PSEs. 

Ci  Life  Support  (LS): 

Life  support  in  Air  Force  systems  development  includes  all  areas  re¬ 
quiring  special  provisions  for  health  promotion,  safety,  protection  and 
sustenance  of  personnel  employed  within  the  total  system  complex. 
Provisions  for  LS  should  be  provided  for  both  normal  and  emerge  .icy 
operations  of  the  system  and  pertain  to  both  aircrew  and  ground  per¬ 
sonnel.  LS  is  devoted  to  the  physiological  and  psychological  well-being 
of  man,  after  his  role  has  been  established  within  the  overall  program. 
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Basie  system  design  will  be  affected  by  the  physiological  or  psycho¬ 
logical  characteristics  of  man.  LS  considerations  will  be  included  in 
the  earliest  conceptual  phase.  As  the  program  progresses,  LS 
requirements,  i.  e. ,  equipment  (protective  and  survival)  and  proce¬ 
dures,  will  be  identified.  An  aggressive  LS  program,  beginning  with 
the  conceptual  phase  and  continuing  through  to  the  establishment  of 
base  health  and  safety  programs,  is  an  Air  Force  requirement. 

(See  AFRs  30-5,  161-2,  and  58-4.) 

d*  Quantitative  and  Qualitative  Personnel  Requirements  Information 
(QQPRI); 

QQPRI  is  personnel  subsystem  data  used  in  planning  for  system  per¬ 
sonnel,  training,  and  manpower,  QQPRI  reports  will  be  developed, 
integrated,  and  published  under  the  direction  of  AFSC  with  the  assist¬ 
ance  of  ATC,  AFLiC,  and  die  operating  command(s).  These  reports 
will  be  time-phased  to  m  *et  the  requirements  of  ATC,  AFLC,  and 
die  operating  commands  as  identified  in  the  System  Master  Schedule. 
QQPRI  changes  will  be  issued  whenever  significant  changes  in  per¬ 
sonnel  requirements  can  be  forecast  during  system /program  develop¬ 
ment  and  test.  QQPRI  is  normally  terminated  at  the  end  of  the 
acquisition  phase  or  early  in  the  operational  phase.  QQPRI  is  com¬ 
posed  of  three  parti,  as  follows; 

(1)  Part  I.  Operations,  Organizational  Maintenance,  and  Field 
Maintenance. 

(2)  Part  II.  Depot  Level  Support.  (Produced  only  on  specific  request 
of  AFLC.) 

(3)  Part  III.  Contractor  Technical  Services  Personnel  Criteria 
(CTSF  Criteria).  This  document  includes  long-range  plans, 
criteria,  and  guidance  for  the  utilization  of  CTSP  in  support  of 
the  operational  employment  of  the  system.  It  is  prepared 
according  to  AFM  66-18. 

e.  Trained  Personnel  Requirements  (TPR); 

TPR  is  a  tabulated  list  of  personnel  requiring  system  peculiar  training 
needed  to  support  a  system  through  the  acquisition  phase  and  is  included 
in  the  personnel  training  section  of  PTDP,  P3PP,  and  SPP.  It  lists 
these  requirements  for  officers,  airmen,  and  civilians  by  AFSC,  grade, 
month  required,  and  command.  Time-phasing  is  based  on  the  date  the 
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individual  is  required  in  place  in  the  development/test,  operational 
and  support  units,  The  TPR  is  initiated  during  the  conceptual  phase 
of  program  development,  and  refined  in  the  PSPP  at  the  conclusion 
of  the  definition  phaae  and  completed  during  the  early  acquisition 
phase  when  QQPRI  documentation  is  approved  by  HQ  USAF,  The  TPR 
ia  basic  to  the  determination  of  training  concepts  and  training  plans, 
and  the  computation  of  training  coats* 

f*  Training  Concepts  x 

j 

The  Training  Concept  which  appears  in  the  PTDP  will  normally  reflect 
personnel  and  training  projections  developed  during  the  conceptual 
phase  and  USAF  guidance  provided  in  SORs  and  specified  ADOs.  In 
cases  where  the  personnel  training  data  is  insufficient,  the  PTDP 
will  indicate  appropriate  requirements  for  personnel  subsystem  efforts 
to  be  accomplished  during  the  program  definition  phase.  The  Train¬ 
ing  Concept  which  appears  in  the  PSPP  reflects  the  combined  efforts 
of  contractors’  responses  to  the  requests  for  proposals  and  work  state¬ 
ments  of  the  program  definition  phase  and  of  in-house  studies  of  data 
available  from  the  conceptual  phase.  Upon  USAF  approval  of  the 
PSPP,  the  Training  Concept  contained  in  section  10  PSPP/SPP  becomes 
the  official  source  document  for  preparing  training  plans  and  will  be 
kept  updated  as  the  system  develops  throughout  the  acquisition  phase. 

g.  Manpower  Authorizations: 

HQ  USAF  (AFOMO)  normally  allocates  manpower  authorizations  by 
Program  Element  Code  (PEC)  whenever  a  system  appears  in  the  force 
structure,  to  fulfill  the  require  .nent  of  the  F&FP  for  5- year  manpower 
data*  Subsequent  PTDP,  PSPP,  SPP  or  QQPRI  documentation  reflect¬ 
ing  manpower  estimates  of  the  participating  commands,  must  verify, 
or  provide  basis  for  change  to  the  initial  authorizations,  as  well  as 
pro/ide  detailed  individual  specialty  i*equirements  such  as  tc  assure 
that  personnel  are  not  trained- out  prior  to  realistic  need  dates. 

h.  System  Manning  and  Trained  Personnel  Requirements  (SMTPR)  Plan: 

(1)  The  SMTPR  plan  will  reflect  the  total  time-phased  system  manning 
and  trained  personnel  requirements.  Problems /solutions  will  be 
shown  in  addition  to  required  special  action  or  deviations,  from 
approved  personnel  policies,  procedures  and  programming  actions. 

HQ  USAF  wUl  use  the  plan  in  determining  availability  of  personnel 
resources  within  the  Air  Force  inventory  for  system  support. 

The  System  Manning  and  Trained  Personnel  Requirements  Plan, 
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as  a  separate  document,  will  be  based  on  the  system  documenta¬ 
tion  referenced  in  AFR  375-4,  data  available  from  operations, 
training  and  logistics  plans,  QQPRI,  and  any  other  pertinent  data, 

(2)  The  SMTPR  plan  will  include  but  not  be  limited  to  the  following: 

(a)  A  listing  of  data  sources  used  such  as  program  and  planning 
documents, 

(b)  Manning  criteria- -command  capability  for  required  AFSCs 
retainability,  oversea  eligibility,  security  clearance. 
Explanation  of  why  and/or  how  existing  personnel  resources 
in  system  or  units  phasing  out  of  the  inventory  should  or 
should  not  be  used  to  meet  the  requirements  of  the  new 
system. 

(c)  Career  Field  Trained  Personnel  Requirements,  tabular  list¬ 
ing  of  officers  and  airmen  by  AFSC  (skill  level  designator 
not  required),  command,  and  fiscal  quarter  required. 

(d)  ORT-- tabular  listing  of  officers  and  airmen  by  AFSC, 
command  z  id  month  required. 

(e)  Time-phased  manning  requirements  by  officers  and  airmen, 
by  AFSC,  command  and  month  reemired  for  input  to  training 
and/or  unit  as  applicable. 

Training  Equipment  Planning  Information  (TEPI): 

TEPI  presents  recommendations  regarding  the  training  equipment  pack¬ 
age  defined  in  AFR  50-19.  Copies  of  TEPI  are  provided  to  ATC,  AFLC, 
and  operating  command(s)  agencies  for  review.  TEPI  provides  a  basis 
for  defining  and  identifying  the  components  of  the  training  equipment 
package  for  inclusion  in  the  PSPP. 

Training  Equipment  Development  (TED): 

TED  includes  all  actions  required  to  define,  program,  budget,  contract, 
develop,  produce,  and  acquire  a  system  training  equipment  package 
defined  in  AFR  50-  19.  The  TED  will  be  defined  during  the  definition 
phase  and  will  be  developed  and  produced  concurrently  with  the  system 
during  the  acquisition  phase  to  insure  timely  availability  to  support 
initial  and  follow-on  training. 
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k*  Training  Facilities: 

Training  Facilities  as  defined  in  this  directive  include  all  real  estate 
and  buildings  to  be  utilized  exclusiveJy  in  support  of  training  programs. 
Depending  upon  the  nature  of  the  system  or  program,  facility  require¬ 
ments  can  range  from  the  simple  identification  of  in- being  classrooms 
to  extensive  modification  of  existing  facilities  and/or  complex  new 
construction  requiring  additions  to  the  Military  Construction  Program 
(MCP).  Tentative  identification  of  training  facility  requirements  will 
be  stated  in  the  PTDP  and  refined  during  the  project  definition  phase 
for  inclusion  in  the  PSPP/SPP, 

!•  Technical  Publications  (TP): 

For  the  purpose  of  this  directive,  TP  are  technical  manuals  of  Tech¬ 
nical  Order  documents  covered  by  AFR  66-7  which  are  needed  to 
support  appropriate  ATC  training  courses.  Appropriate  technical 
publications  are  required  at  the  time  specified  and  in  quantities  desig¬ 
nated  by  the  training  agencies  to  support  training  on  a  given  system, 
subsystem,  or  minor  items  of  equipment. 

m.  Training  Plans: 

Career  Field  and  ORT  training  plans  are  outgrowths  of  the  Training 
concepts  contained  m  the  training  documentation  (PTDP/PSPP). 

These  plana  will  set  forth  in  detail  and  methods,  tim*.- phasing  and 
requirements  necessary  to  accomplish  training  in  support  of  a  system 
or  program.  Although  the  objectives  of  career  field  and  ORT  differ, 
the  plans  for  each  type  of  training  will  be  coordinated  to  prevent 
unnecessary  duplication  of  training/ training  equipment  and  schedule 
corflicts  which  could  jeopardize  programmed  system  test  and  opera¬ 
tional  dates.  The  schedules  for  training  plan  preparation  and  appioval 
by  Headquarters  ATC,  the  preparation  and  approval  ox  ORT  training 
plans  by  the  operating  command(s) /agencies,  and  known  start  and 
completion  dates  of  all  training  plans  will  be  reflected  in  the  master 
schedule. 

n.  Personnel  Subsystem  Test  and  Evaluation  (PSTE): 

In  keeping  with  AFR  80-14,  the  following  policy  and  procedures  will 
apply  to  the  test  and  evaluation  of  the  PS  through  Category  III  test. 
Formal  coordinated  testing  will  start  during  Category  I  testing  and  con¬ 
tinue  through  Category  III  testing  until  it  is  verified  that  the  system 
can  be  operated,  maintained,  and  supported  by  USAF  personnel  in  its 
intended  operational  environment.  To  insure  effective  e validation,  con¬ 
sistent  with  overall  system  test  objectives,  a  coordinated  PSTE  plan 
will  be  identified  in  the  approved  test  plan  for  each  system* 
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APPENDIX  IV 


TESTING/EVALUATION  OF  AIR  FORCE  SYSTEMS, 
SUBSYSTEMS,  AND  EQUIPMENTS;  AIR  FORCE 
POLICY  AND  REQUIREMENTS  (SOURCE;  AFR  30-14) 


A>  Objectives  of  Testing/Evaiuation _ 

In  general,  the  objectives  of  testing  and  evaluation  are  to; 

a.  Measure  and  assess  accomplishment  of  development  objectives, 

b*  Assure  that  systems  and  equipment  meet  established  requirements. 

c.  Obtain  a  true  indication,  forecast,  or  verification  of  the  actual  perform¬ 
ance  capabilities  of  any  given  system,  subsystem,  or  item  of  equipment 
in  as  realistic  an  operatioral  environment  as  practicable. 

d.  Insure  through  effective  testing  timely  integration  of  operationally  ready 
systems  and  support  items  into  the  active  inventory  in  a  logical  order 
from  conceptual  phase  through  acquisition  into  the  operational  phase. 

e.  Detect  operational  and  engineering  deficiencies  in  time  for  changes  ter 
be  incorporated  prior  to  significant  production  bui]d-up.  Insure  that 
changes  to  operational  equipment  meet  the  required  objectives  or  that 
necessary  trade-offs  are  identified. 

f .  Maintain  and  enhance  organic  capability  for  evaluation  of  systems  and 
equipment  developed  to  meet  Air  Force  needs. 

g.  Provide  testing  services  for  Ai •  Force  agencies  and  non- Air-Force 
agencies  requiring  technical  krowledge,  capabilities,  or  facilities  which 
exist  primarily  within  USAF  research  and  development  agencies. 

h.  Provide  data  and  operational  nnalyses  for  application  to  current  and  fu¬ 
ture  systems  and  system  studies. 

i.  Evaluate  the  over-all  logistic  capability,  scope,  and  effectiveness  as  pre¬ 
scribed  by  appropriate  support  procedures,  plans,  and  planning  factors, 
developed  concurrently  with  system  evolution.  Acquire  and  evaluate  data 
to* 

(1)  Verify  and  refine  logistic  procedures,  plans,  and/or  planning  factors. 
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(2)  Enhance  prospective  support  planning. 

(3)  Identify  areas  which  will  require  additional  impetus  to  insure  inte¬ 
grated,  effective  aystem(s)  logistic  support. 

j,  Identify  and  assess  manpower  spaces  and  personnel  resources  necessary 
to  support  systems  and  equipment. 

k.  Provide  training  planning  information,  technical  information,  and  task  an¬ 
alysis  data  for  the  purpose  of  validating  training  and  training  programs. 

1*  Integrate  hardware,  software,  and  manpower  into  an  entity  that  meets 
requirements  most  effectively  (primarily  for  !,LH  systems). 

_ _ B .  Policies  and  Procedures  _ _ 

a.  In  no  instance  will  tests  be  conducted  without  evaluation.  However, 
evaluation  may  be  conducted  without  current  testing  because  of  data 
available  from  earlier  tests  or  other  sources  which  may  suffice,  thus 
saving  time  and  resources. 

b.  Tes*  and  evaluation  will  be  documented  and  treated  as  a  part  of  the  sys¬ 
tem  acquisition,  research  and  development  program,  operational  test, 
or  engineering  service  effort  with  which  it  is  identified. 

c.  Consideration  will  be  given  to  the  use  of  available  commercial  items  in 
lieu  of  developmental  items,  wherever  practical. 

d.  New  and  modified  systems  or  support  items,  including  nuclear  weapons 
and  associated  equipment,  will  undergo  test  and  evaluation  during  acqui¬ 
sition  to  determine  if  they  meet  established  requirements  and  are: 

(1)  Technically  sound,  reliable,  and  safe  for  service  use. 

(2}  Functionally  operable,  reliable,  maintainable,  and  compatible  with 
other  systems  or  equipment  in  the  environment  in  which  the  items 
will  be  employed. 

(3)  Able  to  be  maintained  with  a  minimum  expenditure  of  resources  con¬ 
sistent  with  operational  requirements,  e.g»,  manpower,  support  and 
test  equipment,  special  tools,  training,  spare  parts,  and  special 
facilities. 

(4)  Capable  of  being  operated  and  maintained  by  Air  Force  personnel 
after  completion  of  prescribed  training. 


(5)  Compatible  with  associated  warheads  and  other  armaments  as 
applicable* 

(6)  Designed  to  be  as  free  as  possible  from  design  features  or  pro- 
cedures  that  will  evoke  personnel  errors  in  operation  or  main¬ 
tenance* 

(7)  Capable  of  being  transported  by  programmed  mode  of  transporta¬ 
tion  and  specific  carrier  equipment*  Requirements  of  AFM  75-2 
will  be  included  as  early  in  the  test  program  as  is  feasible  unless 
specifically  exempted  by  HQ  USAF* 

(8)  Designed  to  permit  maximum  ease  of  accessibility  to  equipment 
requiring  replacement,  servicing,  adjustment,  or  calibration* 

e*  Test  programs  during  the  acquisition  phase  (Category  I  and  II)  of  a  giwn 
series  of  a  system  or  item  of  equipment  will  be  conducted  under  the 
control  of  the  AFSC  System /Equipment  Program  Director  as  appropriate. 
System  tests  will  be  performed  by  a  test  force  or  forces  at  one  or  more 
locations  which  include  all  agencies  involved  in  the  system* 

f.  Operational  testing  and  evaluation  will  be  scheduled  for  all  systems, 
applicable  subsystems,  and  operational  support  items  subject  to  status 
classification  (AFR  80-6).  Operational  support  items  developed  in 
support  of  a  validated  Operational  Support  Requirement  (GSR)  will  be 
tested  by  AFSC  prior  to  operational  testing  by  an  Operating  Command 
(AFRS  57-3,  80-2).  Except  for  Strategic  Ballistic  Missile  Testing 
(see  paragraph  7),  operational  testing,  but  not  evaluation,  maybe 
waived  by  the  Operating  Command  if  sufficient  test  data  are  obtainable 
from  other  sources. 

g*  Follow-on  developmental  tests  may  be  required  when  significant  changes 
are  made  in  system  capability,  new  subsystems  or  components  are 
added,  a  system  is  integrated  into  a  new  environment,  or  changes  to 
correct  deficiencies  are  made,  and  test  articles  are  not  available  dur¬ 
ing  the  normal  test  cycle;  or  when  analysis  of  test  data  dictates  verifi¬ 
cation  of  previously  conducted  tests.  Provisions  for  such  testing  must 
be  considered  throughout  the  acquisition  process  (Category  I  and  II)  and 
planned,  programmed,  an?  executed  with  the  priority  and  emphasis 
afforded  the  overall  system  * 

h.  Consideration  will  be  given  to  joint  experimental  testing  with  other 
Services  (Army,  Navy,  and  Marine  Corps)  in  order  to  insure  that  Air 
Force  systems  are  capable  of  operating  in  a  joint  environment. 
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U  Maximum  use  will  be  made  of  existing  test  facilities  and  capabilities 
which  can  be  made  available  by  AF3G,  the  contractors),  the  par¬ 
ticipating  commands,  and  other  Government  agencies.  The  most 
realistic  operational  environment  attainable  will  be  used  for  develop¬ 
ment  and  operational  testing.  Testing  will  be  consolidated  when 
feasible  to  avoid  duplication.  Maximum  use  will  be  made  of  test 
data  available  from  other  sources  or  obtained  during  early  stages  of 
development  testing, 

j*  Provisions  will  be  made  for  early  and  progressively  increasing  Operat¬ 
ing  and  Supporting  Command  participation  in  system  and  subsystem 
development  test  and  evaluation.  This  is  necessary  to  provide 
familiarisation,  training,  and  experience  required  to  achieve  the  ear¬ 
liest  operational  and  logistic  support  capability.  Included  i  re  Engi¬ 
neering  Inspection  Boards  (AFR  80-28)  and  Configuration  Control 
Boards  (AFRS  57-4  and  65-3). 

k.  Syatems  and  items  allocated  for  the  test  inventory  should  be  produced 
as  rapidly  as  possible  to  permit  adequate  testing.  Necessary  changes 
disclosed  by  testing  will  be  incorporated  into  production  articles  at 
the  earliest  practicable  date. 

l,  Allocation  and  delivery  of  test  equipment,  test  support  equipment,  or 
spares  for  systems  undergoing  test  (Category  I  and  II)  will  huve 
precedence  over  production,  training,  or  operational  requirements 
for  all  equipment  or  personnel,  except  when  otherwise  directed  by  HQ 
USAF. 


C.  System  Testing  and  Evaluation  _ _ 

System  test  programs  will  normally  be  conducted  in  two  functional  categories 
during  the  acquisition  phase,  and  one  category  during  the  operational  phase 
(except  for  Strategic  Ballistic  Missiles).  These  are: 

a*  Category  I--5ubsystem  Development  Test  and  Evaluation-- consists  of 
development  testing  and  evaluation  of  the  individual  components,  sub¬ 
systems,  and,  in  certain  cases,  the  complete  system.  In  addition  to 
qualification,  the  testing  provides  for  redesign,  refinement,  and 
re-evaluation  as  necessarv,  including  the  practicality  of  utilizing  cur¬ 
rent  standajd  and  commercial  items.  These  tests  are  conducted  pre¬ 
dominantly  by  the  contractor  but  with  Air  Force  participation,  evalua¬ 
tion,  and  control  exercised  through  AFSC.  Specific  examples  of  test 
objectives  are  the  determination  of: 

(I)  Performance,  reliability,  and  integrity  of  individual  components. 
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(2)  Compatibility,  reliability,  *nd  adequacy  of  Government  Fur¬ 
nished  Equipment  (GFE),  Government  Furnished  Aerospace 
Equipment  (GFAE),  commercial  equipment,  or  standard  Air 
Force  Kerrv  for  incorporation  into  the  system. 

(3)  Preliminary  operating  characteristics  and  qualitative  adequacy 
of  the  system,  subsystems,  and  components. 

(4)  Prelimim  ry  performance,  stability,  control  characteristics, 
and  general  airworthiness  (as  appropriate)  of  the  aerospace 
vehicles,  or  similar  criteria  determination  for  electronic  and 
support  sy&  cems  / equipment. 

(5)  Preliminary  compatibility,  adequacy,  supportability,  and  relia¬ 
bility  of  Aerospace  Ground  Equipment;  Ground  C-E-M;  and 
commercial  ground  CUE  and  computer  equipment. 

(6)  Preliminary  maintainability  and  transportability  characteristics 
of  components  and  subsystems. 

(7)  Preliminary  validity  of  personnel  and  training  planning  informa¬ 
tion  used  for  personnel  skill  identification  and  development  of 
manning  documents,  training  and  training  equipment  require¬ 
ments.  Additional  requirements  to  insure  personnel  and  training 
support  will  also  be  determined.  Whenever  feasible,  formal 
training  will  be  evaluated  (see  AFR  30-8), 

(8)  Preliminary  identification  and  investigation  of  safety  criteria 
for  explosive  ordnance  and  any  safety  problems  which  must  be 
resolved  before  initial  operational  capability  is  established, 

(9)  Necessary  data  for  preliminary  handbooks /technical  manuals, 

(10)  Preliminary  evaluation  of  the  overall  logistic  plans,  policies, 
and  procedures  to  insure  consonance  with  the  logistic  concept. 

(11)  Evaluation  of  new  design  or  updating  changes, 

(12)  Adequacy  of  preliminary  health  hazards  data  and  precautionary 
information. 

(13)  Procedures  for  prevention  of  and/or  recovery  from  potentially 
catastrophic  situations  (missile  and  space  launches,  etc.). 
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b*  Category  R==Syst©m  Development  Test  and  Evaluation-* consists  of 
testing  and  evaluation  spanning  the  integration  of  subsystems  into  a 
complete  system  in  as  near  an  operational  configuration  as  is  prac¬ 
ticable*  Suitable  instrumentation  will  be  employed  to  determine  the 
functional  capability  and  compatibility  of  subsystems.  Category  II 
is  a  joint  contractor- Air  Force  effort  under  Air  Force  control  during 
which  the  Air  Force  effort  becomes  predominant  with  ever-increasing 
operating  and  supporting  command  participation.  Actual  test  opera  * 
tion  and  maintenance  should  be  performed  by  military  perso>  nel  who 
have  received  formal  system  training.  It  is  usually  culminated  with 
th^  demonstration  effort  required  to  complete  the  development  por¬ 
tion  of  the  acquisition  phase  of  a  system  program.  Specific  examples 
of  test  objectives  are? 

(1)  Determine  that  the  system /equipment  meets  established  require¬ 
ments  and  specifications  for  performs  ce,  control,  maintenance, 
safety,  reliability,  etc, 

»)  Define  the  operational  configuration, 

(3)  Determine,  develop,  and  test  updating  changes  that  are  necessary 
to  meet  approved  performance  requirements, 

(4)  Refine  logistic  procedures  and  policies, 

(5)  Verify  and  validate  required  technical  data.  This  term  is  inter¬ 
preted  in  its  broadest  scope  and  will  include  prints,  drawings, 
handbooks,  manuals,  technical  documents,  and  other  related 
publications, 

(6)  Evaluate  new  design  changes  before  incorporation  into  the  pro¬ 
duction  system, 

(7)  Determine  capabilities,  limitations,  and  safety  characteristics, 
under  actual  or  simulated  climatic  conditions  by  ground  and/or 
flight  tests  (as  appropriate).  These  tests  will  Vs  designed  to 
yield  both  engineering  and  handbook  data, 

(S)  Provide  familiarisation,  experience,  and  limited  training  to 
Supporting  and  Operating  Command  personnel.  However,  pri¬ 
mary  test  objectives  will  remain  paramount. 

(9)  Demonstrate  in  the  most  realistic  environment  practicable  that 
the  complete  system  is  operable,  maintainable,  and  transportable 
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(as  appropriate)  with  approved  and  minimum  support  and  test 
equipment,  personnel,  training,  special  tool*,  spare  part*, 
technical  data,  and  special  facilities. 

(10)  Determine  the  adequacy  of  the  Personnel  Subsystem  (AFR  30-8) 
and  accelerate  actions  when  changes  in  personnel  training  and/or 
manning  for  the  system  in  its  operational  environment  are  re¬ 
quired.  Verify  that  personnel  subsystem  performance  is  ade¬ 
quately  supported  by  equipment  design,  tools,  technical  data, 
job  environment,  training,  personnel  selection,  manning  and 
organisation  control  procedures. 

(11)  The  Category  II  Test  and  Evaluation  shall  not  be  considered 
complete  until  one  of  the  following  has  taken  place; 

fa)  Performance  requirements  as  directed  by  HQ  USAF 

(Specific  Operational  Requirement,  Development  Directive, 
or  Operational  Support  Requirement,  etc, ,  as  applicable) 
have  been  met,  and  it  has  been  demonstrated  that  qualified 
Operating  Command  personnel  can  effectively  prepare, 
operate,  and  maintain  the  system  utilizing  only  authorized 
equipment  and  technical  procedures  and  data. 

(b)  HQ  USAF  has  officially  relieved  AFSC  of  performance  re- 
quiremiuts  that  have  been  recommended  for  deletion  due 
to  advantageous  trade-offs  of  cost  or  program  schedule. 

(c)  The  Operating  Command  and  AFSC  both  agree  there  are 
minor  areas  of  performance  remaining  that  cannot  be 
demonstrated  within  a  reasonable  time  period.  These  may 
be  caused  by  such  things  as  long  lead  time  components,  or 
recent  engineering  changes  that  are  not  available,  or  ex¬ 
tended  tests  and  evaluations  that  rely  on  long  time  periods 
of  data  gathering,  etc,  A  formal  agreement  must  be  ren¬ 
dered  among  the  Operating  Command(s),  AFSC,  AFLC,  and 
ATC  defining  the  uncompleted  areas  and  showing  how  they 
will  eventually  be  tested, 

c.  Category  III- -System  Operational  Test  and  Evaluation  Progra  -  --con¬ 
sists  of  test  and  evaluation  of  operational  systems  under  the  t*  :trol 
and  direction  of  die  Operating  Command.  These  tests  shall  include 
all  components,  support  items,  persons  *  skills,  technical  data,  and 
procedures  and  shall  be  performed  under  as  near  operational  condi¬ 
tions  as  practicable.  Suitable  instrumentation  will  be  employed  in 
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order  t©  adequately  evaluate  t m%  result*.  Category  III  testing  will 
bt  tondueted  utilising  a  configuration  as  jointly  agreed  by  the  Opera t= 
ing  Cumnrand  and  AFSC/AJ?  LG,  The  test  will  be  conducted  in 
accordance  with  a  specific  te^t  plan  or  order  designed  to  meet  the 
objectives  of  all  participants.  The  test  force  sise  and  composition 
will  be  as  mutually  agreed  in  the  Operational  Test  and  Evaluation  Plan 
or  Order,  Accomplishment  may  be  performed  at  Operating  Com* 
mand,  AFSC,  or  other  available  installations  as  circumstances  dic¬ 
tate,  Specific  examples  of  test  objectives  arc; 

(1)  Determine  and  improve  the  operational  capabilities  of  the  sys¬ 
tem  and  develop  the  most  effective  operational  tactics,  cech- 
niques,  doctrine,  and  standards, 

(2)  Determine  any  operational  deficiencies  and/or  limitations  and 
provide  quantitative  and  qualitative  data  for  product  improvement 
programs, 

(3)  Evaluate  the  logistic  system  and  capability.  Acquit  c  supplemental 
logistic  data  on; 

(a)  The  rate  of  parts  consumption,  maintenance  and  support 
facility  requirements  obtained  during  previous  tests, 

(b)  Adequacy  of  off-the-shelf  equipment,  as  applicable; 

(c)  Supportability  of  commercial  equipment  incorporated  in  the 
system; 

(d)  Compatibility  of  equipment  and  components  with  transporta¬ 
tion  equipment  planned  for  use  in  support  of  the  equipment. 

(4)  Determine  the  adequacy  of  trained  per^nnel  to  operate  and  main¬ 
tain  the  system  in  it*  operational  environment.  Take  action  to 
re/isc  the  personnel  subsystem  and  related  training  programs 
and/or  manpower  documents  and  authorizations  when  validated 
test  results  so  indicate. 

(5)  Obtain  supplemental  data  relative  to  operational  and/or  support 
requirements  in  terms  of  personnel;  maintenance;  supply; 
transportation;  packaging  and  materials  handling;  training; 
special  tools,  test  and  support  eqiipment;  special  facilities; 
general  performance  standards. 
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(6)  Determine  the  adequacy  of  technical  data  and  provide  informa¬ 
tion  for  early  corrections  or  additions,  as  found  necessary. 

(7)  Supplement  and  refine  reliability  data. 

(8)  Determine  the  adequacy  of  health  protection,  life  support,  and 
medical  and  safety  procedures,  directives  and  equipment. 

(9)  Verify  configuration  status  and  reporting  system  between  AFLC 
and  supporting  command  where  mechanized  system  of  accounting 
is  used. 

(10)  Insure  that  acceptable  reliability  factors  are  attained. 

_  D.  R&D  Project  Test  and  Evaluation  _ 


a.  Test  and  Evaluation  involved  in  the  Research  and  Development  program 
(Research,  Exploratory  Development,  Advanced  Development,  Engi¬ 
neering  Development,  and  certain  Management  and  Support  items)  will 
be  administered  and  conducted  as  an  integral  part  of  research  or 
development  projects  with  which  identified.  Criteria  by  program  area 
are: 

(1)  Operational  Support  (normally  a  segment  of  engineering  develop¬ 
ment).  Testing  and  evaluation  associated  with  operational  support 
items  from  relatively  simple  development  and  operational  testing 
of  minor  support  items  to  large-scale  efforts  equalling  those 
involved  in  system  acquisition. 

(a)  Testing  of  Operational  Suppo  t  items  will  include: 

1.  Development  Testing  and  E\'aluation  to  achieve  objectives 
similar  to  those  noted  in  Category  I  and  II  system  develop¬ 
ment  testing,  as  appropriate  for  operational  support 
development. 

2.  Operational  Test  and  Evaluation  to  achieve  objectives 
similar  to  those  noted  in  Category  III  system  operational 
testing,  as  appropriate- for  operational  support  development. 

(b)  Subsequent  to  the  completion  of  development,  test,  and  status 
classification  of  an  operational  support  item,  it  may  be  speci¬ 
fied  for  application  to  one  or  more  new  systems  under  develop¬ 
ment.  Additional  testing  to  determine  compatibility  in  the 
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new  system  environment  may  be  required  and  is  not 
considered  duplicative. 

(2)  Advanced  Development-Engineering  Development  (other  than 

Operational  Support)  and  Test  Instrumentation: 

(a)  Development  Testing  and  Evaluation  will  be  performed  to 
achieve  objectives  of  the  type  specified  for  Category  I  and 
II  system  development  testing,  as  appropriate  to  advanced 
subsystem,  component,  and  test  instrumentation  development. 

(b)  An  outline  of  testing  and  experimentation  necessary  to  ac¬ 
complish  the  proposed  development  will  be  included  in  the 
planning  documentation  specified  in  AFR  80-27.  As  with 
Explozatory  Development  and  Research  projects,  state¬ 
ments  of  resources  and  facility  requirements  will  be 
provided. 

(3)  Exploratory  Development  and  Research  Projects: 

(a)  Research  Testing  or  Experimentation  will  be  performed  to 
verify  hypotheses  and  proposed  solutions  to  operational 

*  needs  or  to  measure  phenomena  in  the  acquisition  of  new 
knowledge,  as  appropriate,  but  not  beyond  the  intent  as 
included  in  AFR's  80-27  and  80-4. 

(b)  An  outline  of  testing  and  experimentation  necessary  to  ac¬ 
complish  proposed  research  will  be  included  in  the  planning 
documentation  specified  in  AFR  80-27.  Resources  and 
facilities  required  for  performing  the  testing  and  experi¬ 
mentation  will  be  included  in  the  documented  plan. 

_  E.  Engineering  Services  Test  and  Evaluation _ 

Testing  services,  provided  to  Air  Force  and  non- Air  Force  agencies  requir¬ 
ing  technical  support,  facilities,  and  knowledge  which  exist  primarily  within 
USAF  research  and  development  agencies  will  be  considered  as  engineering 
services.  These  services,  which  are  not  identifiable  with  an  approved  Air 
Force  research  and  development  program,  will  be  provided  in  accordance 
with  applicable  support  agreements  or  the  importance  of  the  task  as  deter¬ 
mined  by  precedence  rating,  national  interest,  or  as  directed  by  HQ  USAF, 
to  support: 
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it.  Operational  engineering  activities,  which  include  11  Unsatisfactory 
Report”  engineering,  modification  engineering,  re- procurement 
support,  and  evaluation  of  inventions  and  techniques, 

b.  Engineering  support  of  other  Air  Force  and  DOD  agencies,  Govern¬ 
ment  (including  AEC  and  NASA)  and  industrial  agencies, 

c.  The  Defense  Standardization  Program  (AFR  73-1). 

______  F.  Strategic  Ballistic  Missile  Testing  _ 

In  addition  to  the  provisions  of  this  regulation,  certain  testing  identifica¬ 
tion  and  terminology  will  apply  exclusively  for  reports  to  JCS/OSD  on 
strategic  ballistic  missile  testing.  As  shown  in  attachment  1,  these 
included  R&D  tests  (equivalent  to  Category  I  and  II),  DASO  tests  (equiva¬ 
lent  to  Category  IH);  plus  Operational  tests  and  Follow-on  Operational 
Tests,  which  are  explained  as  follows: 

a.  Operational  Tests.  Testing  and  evaluation  of  systems  reliability  for 
Single  Integrated  Operations  P!an  (vSIOP)  of  operational  systems  com¬ 
mitted  to  the  SIOP.  The  objectives  of  these  tests,  conducted  by  the 
Operating  Command,  are  to  det  :rmine  the  readiness,  launch,  inflight 
reliability,  and  accuracy  of  each  weapon  system.  A  sufficient  nurrber 
of  tests  will  be  conducted  to  achieve  the  objectives  with  the  confidence 
levels  as  directed.  Operational  Tests  will  bp  accomplished  subse¬ 
quent  to  DASO  tests  and  will  be  conducted  in  as  near  an  operational 
environment  as  possible.  Units  and  missiles  tested  will  be  represen¬ 
tative  of  each  respective  missile  force.  Specific  objectives  will  be 
identified  in  a  Test  and  Evaluation  Plan  (for  each  weapon  system)  sub¬ 
mitted  to  HQ  USAF  for  approval. 

b.  Follow-on  Operational  Tests.  These  tests  consist  of  continuing  test¬ 
ing  and  evaluation  by  the  Operating  Command.  The  objectives  of  these 
tests  are  to  insure  that  the  reliability  of  each  respective  weapon  sys¬ 
tem  is  preserved  and  planning  factors  remain  valid  for  continued  SIOP 
use.  Any  additional  objectives  will  be  identified  in  the  Test  and  Evalua¬ 
tion  Plan  (for  each  weapon  system)  submitted  to  HQ  USAF  for  approval. 
These  tests  will  sample  each  ballistic  missile  weapon  system  on  an 
annual  basis.  Tests  will  be  conducted  in  sufficient  number  under  an 
operational  environment  representative  of  the  missile  force  to  reveal 
any  change  in  weapon  system  performance. 
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APPENDIX  V 


COGNIZANT  PERSONNEL  WITH  WHOM  DISCUSSIONS 
WERE  HELD  DURING  THE  SURVEY 


Name 


Affiliation 


J.  Adams  SEMP 

Wright-Patterson  AFB,  Ohio 


Capt,  T*  Aldrich,  USAF 
C*  Bates 

L.  Becket 

H.  Bcr ridge 
C.  Bishop 

J.  Blank 

J.  Booth 

K.  Borchers 

W.  Chase 

B*  Cohen 


SEMP 

Wright-Patterson  AFB,  Ohio 

Behavioral  Sciences  Laboratory 
Wright- Patter  son  AFB,  Ohio 

General  Dynamics  Corp. 

Ft.  Worth,  Texas 

Air  Proving  Ground  Center 
Eglin  AFB,  Florida 

416M  System  Program  Office 
L.  G.  Hanscom  Field 
Bedford,  Mass, 

The  Boeing  Company 
Vandenberg  AFB,  California 

The  Boeing  Company 
Seattle,  Washington 

Space  Technology  Laboratories 
Los  Angeles,  California 

Space  Technology  Laboratories 
Los  Angeles,  California 

MacDonnell  Aircraft  Corporation 
St.  Louis,  Missouri 


J»  Coules 


Decision  Sciences  Laboratory 
L.  G.  Hanscorr  Field 
B edford,  Mas  sachusetts 


J  27 


Nam* 


Affiliation 


G,  Grit** 

Lt,  Col*  G*  Crosier,  USAF 

C*  Do  vino 

J,  Donaldson 

J,  Dorton 

J.  Edelman 

G.  Eckstrand 
Mildred  English  . 

G.  Evans 

B.  Fulk 

Gloria  Grace 

C.  Gustafson 

Capt,  J.  Harris,  USAF 

G.  Hayes 


MaeDonncll  Aircraft  Corp* 

St,  Louis,  Missouri 

ATTWP 

Wright~Patteraon  AFB,  Ohio 

Satellite  Control  Facility 
Sunnyvale,  California 

The  Martin  Company 
Denver,  Colorado 

MRPTP 

Wright-Patterson  AFB,  Ohio 

Grumman  Aircraft  Engineering  Corp. 
Bethpage,  Long  Island 

MRPT 

Wright-Patterson  AFB,  Ohio 

The  Boeing  Company 
Seattle,  Washington 

General  Dynamics  Corp. 

Ft.  Worth,  Texas 

General  Dynamics  Corp. 

Ft.  Worth,  Texas 

System  Development  Corp. 

Santa  Monica,  California 

SEPSM-RTD 

Wright-Patterson  AFB,  Ohio 

C-141  Personnel  Subsystem 
Test  Office 

Edwards  AFB,  California 

Lockheed  Aircraft  Co. 

Marietta,  Georgia 
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Name 
S,  Heekert 

J,  Hyde 

A,  Jeffers 

E,  Jones 

Lt,  Col,  W.  Jones,  USAF 

Capt.  H,  Kagan,  USAF 
S,  Kaplan 

Lt,  C.  Kreunen,  USAF 

L.  La  Porte 

R.  Leadingham 

E.  Levin 

J.  Mangle sdorf 

R.  Martel 


Affiliation 

behavioral  Sciences  Laboratory 
Wright-Patterson  AFB,  Ohio 

The  Martin  Company 
Denver,  Colorado 

SEMZ-1 

Wright-Patterson  AFB,  Ohio 

MacDonnell  Aircraft  Corp, 

St,  Louis,  Missouri 

Titan  II  System  Program  Office 
Norton  AFB 

San  Bernadino,  California 

Space  Systems  Division  (3SOT) 
El  Segundo,  California 

SEMP 

Wright-Patterson  AFB,  Ohio 

C-141  Personnel  Subsystem 
Test  Office 

Edwards  AFB,  California 

Autonetics  Division 
Ford  Motor  Company 
Downey,  California 

MacDonnell  Aircraft  Corp. 

St.  Louis,  Missouri 

Grumman  Aircraft  Corp. 
Befchpage,  Long  Island 

Lockheed  Aircraft  Company 
Sunnyvale,  California 

Syl vania  Electronics  Corp. 
Waltham,  Mass. 
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Namt 
E*  Martin 

W*  MeAbce 

C*  McLean 

Moore 

E.  Miller 
Sara  Munger 
Si  Murdock 

W,  Oberthorn 

H.  Organ 

H.  Ozkaptan 

Maj .  J,  Reed,  USAF 


Affiliation 

Lockheed  Aircraft  Company 
Marietta,  Georgia 

General  Dynamics  Corp* 

Ft,  Worth,  Texas 

SEMP 

W right-Patte r s on  AFB,  Ohio 

MacDonnell  Aircraft  Corp, 

St,  Louis,  Missouri 

The  Martin  Company 
Denver,  Colorado 

American  Institutes  for  Research 
Washington,  D.  C, 

Boeing  Company 
Seattle,  Washington 

Grumman  Aircraft  Engineering  Corp, 
Ft.  Worth,  Texas 

Satellite  Control  Facility 
Sunnyvale,  Calif o r nia 

Grumman  Aircraft  Engineering  Corp, 
Bethpage,  Long  Island 

Satellite  Control  Facility 
Sunnyvale,  California 


Maj,  S.  Reed,  USAF  Space  Systems  Division  (AFSC) 

Inglewood,  California 

E.  Rieck  SEMP 

Wright-Patterson  AFB,  Ohio 

Ring  MRO 

Wright-Patterson  AFB,  Ohio 
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Name 
r,  Roberts 

Ll,  D,  Kook,  USAF 

Capt,  C,  Scoggins,  USAF 

M,  Snyder 
P,  Sprey 

LI,  Col,  W.  Stobie,  USAF 

C.  Stout 

R.  Turner 

B.  Wolin 


Affiliation 

System  Development  Corp. 

Bedfcrd,  Mess, 

Minuteman  System  Program  Office 

Norton  AFB 

San  Bernadino,  Calif, 

Electronic  Systems  Davit  von  (AFSC) 
L,  G,  Han^rom  Field 
Bedford,  Massachusetts 

Behavioral  Sciences  Laboratory 
Wright-Patterson  AFB,  Ohio 

Grumman  Aircraft  Engineering  Corp. 
Ft.  Worth,  Texas 

Air  Proving  Ground  Center 
EglinAFB,  Florida 

Titan  II  System  Program  Office 
Norton  AFB 

San  Bernadi 'o,  California 

The  Bocung  Company 
Seattle,  Washington 

Systems  Development  Corp. 

Santa  Monica,  California 
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This  report  describes  the  current  practices  and  evaluation  aspects  of  human  per-  * 
formance  assessment  in  Air  Force  Systems,  The  human  performance  test  programs 
for  thirty-four  systems  and  subsystems  representing  the  major  types  of  systems 
(aeronautical,  electronic,  missile,  and  space)  used  by  the  Air  Force  are  reviewed. 
For  these  systems,  the  major  functional  areas  covered  include:  (1)  Air  Force 
policies,  directives,  requirements,  and  constraints  concerning  the  development 
and  assessment  of  system  tests  and  human  performance;  (2)  the  behavioral  sciences 
approach  to,  and  technology  for,  assessing  human  performance;  and  (3)  Air  Force 
practices  in  assessment  of  human  performance.  Throughout,  the  systems  context, 
within  which  human  performance  is  conceived  and  evaluated,  is  emphasized.  Con¬ 
sequently,  the  techniques  within  the  behavioral  sciences  for  examining  human 
performance  conceptually  and  empirically  in  the  system  test  environment  is  a 
particularly  practicable  part  of  the  report,.  The  report  is  supported  by  many  useful 
tables  and  charts,  excerpts  from  test  directives  pertinent  to  human  performance  ! 
assessment,  and  approximately  600  categorized  references. 
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